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EXECUTIVE  SUMMARY 


Air  traffic  controller  decision  making  is  a  key  contributor  to 
safety  in  the  National  Airspace  System  (NAS) .  Insufficient 
systematic  attention  has  been  paid,  however,  to  investigating  the 
parameters  of  controller  decision  making,  or  the  factors  that 
affect  decision  making  in  the  air  traffic  control  (ATC) 
environment.  Because  controller  decision  making  takes  place  within 
the  context  of  the  controlled  airspace,  it  is  necessary  to 
understand  the  elements  of  that  context  as  potential  influences  on 
decision-making  outcomes.  The  research  documented  in  this  report 
was  designed  to  investigate  the  elements  of  en  route  sector 
complexity  in  the  Southeast  Region,  and  to  examine  the  effects  of 
those  elements  on  controller  decision  making. 

StVdy  9f -SggjfeQE- COPPlg-Xity. 

Phase  I  of  the  research  focused  on  collecting  and  analyzing  data  on 
sector  complexity.  The  research  took  place  at  the  Jacksonville  Air 
Route  Traffic  Control  Center  (ARTCC)  in  Jacksonville,  Florida.  Two 
complementary  methods  were  used:  1)  direct  elicitation  of  sector 
complexity  factors  and  related  data  from  Jacksonville  personnel; 
and  2)  indirect  identification  of  sector  complexity  factors  through 
the  application  of  statistical  procedures. 

A  preliminary  study  consisted  of  tests  of  data  collection  and 
analysis  methods,  and  produced  a  list  of  24  initial  complexity 
factors  for  further  study.  The  goals  of  the  main  study  were  to 
apply  the  data  collection  methods  to  a  larger  group  of  controllers, 
develop  an  initial  set  of  complexity  factors,  and  verify  that  a 
reduced  set  of  these  factors  could  adequately  account  for  sector 
complexity.  Analysis  of  the  data  from  the  main  study  identified  a 
set  of  19  Initial  Factors  that  were  used  for  further  analysis.  To 
complete  Phase  I,  11  Final  Factors,  that  showed  a  significant 
relationship  with  sector  complexity,  were  identified. 

Other  results  of  Phase  I  indicated  that  individual  controllers  may 
respond  differently  to  given  constellations  of  complexity  factors. 
Comparison  of  the  complexity  factors,  generated  by  the  direct  and 
indirect  methods,  suggested  that  the  direct  method  can  be  used 
reliably  to  obtain  information  on  sector  complexity.  Although  the 
indirect  method  provides  comparable  information,  the  time  spent  in 
interpreting  the  results  adds  to  the  cost  of  the  analysis. 

A  comparison  was  made  between  the  ordering  of  Jacksonville  sectors 
that  emerged  from  participants'  factor  ratings,  and  the  ordering 
from  a  previous  annual  review  that  used  a  Federal  Aviation 
Administration  (FAA)  formula  for  determining  sector  complexity. 
Although  the  two  orders  were  statistically  related,  the 
relationship  was  an  extremely  weak  one.  It  is  possible,  therefore, 
that  the  FAA  rating  system  may  not  adequately  account  for  sectoz 
complexity.  Further  validation  work  is  needed,  however,  to  confirm 


the  improved  effectiveness  of  the  method  for  evaluating  sector 
complexity  developed  by  this  study. 

Study  of  Controller  Decision  MaKino. 

Phase  II  of  the  study  was  designed  to  investigate  controller 
decision  making  within  the  context  of  the  sector  environment.  ATC 
simulation  problems  were  developed  to  reflect  sector  complexities 
ranging  from  low  to  high.  One  problem  was  used  in  a  preliminary 
study.  Before  working  the  problem,  participants  were  asked  to 
predict  the  ATC  conflicts  that  would  occur,  to  describe  anticipated 
control  actions  along  a  timeline,  and  to  predict  the  trajectories 
of  aircraft  in  the  problem.  They  also  rated  the  problem  on  the 
sector  complexity  factors  identified  in  Phase  I. 

Results  for  Phase  II  are  related  to  several  key  categories  of 
controller  behavior:  problem  analysis,  planning,  problem  solving, 
and  acting.  An  interesting  result  for  problem  analysis  is  that 
participants  did  not  agree  on  the  anticipated  number,  or  safety 
risk,  of  potential  conflicts  in  the  problem.  Some  participants 
predicted  more  conflicts  than  would  have  actually  occurred,  while 
some  did  not  predict  those  that  would  have  occurred. 

The  effectiveness  of  participants'  planning  activities  was  assessed 
by  comparing  their  predictions  to  their  actual  actions.  The 
effectiveness  of  the  initial  plan  varied  from  16  to  73  percent. 
Participants  were  able  to  draw  the  projected  routes  of  aircraft 
through  to  the  end  of  the  problem  and  in  general,  planned  flight 
paths  closely  resembled  the  actual  routes  taken. 

Evidence  was  found  for  Recognition->Primed  Decision  Making  (RPD) , 
that  is,  controllers  who  were  familiar  with  the  sector  and  the 
problem,  and  recognized  the  problem  and  the  solution.  Most 
participants  combined  RPD  with  working  out  the  details  of  the 
solution.  This  finding  converges  with  other  research  on  decision 
making  that  has  been  conducted  in  other  problem  domains  (Klein, 
1989} . 

The  nvunber  of  actions  taken  by  the  participants  ranged  from  21  to 
32.  Reasons  for  changes  in  the  plan  included  unpredictable 
aircraft  behavior,  detection  of  conflicts,  and  changes  in  the 
sequencing  method  employed. 

Analysis  of  the  effects  of  sector  complexity  factors  indicated  a 
lack  of  agreement  between  the  participants  on  some  of  the 
complexity  scales.  The  highest  rated  complexity  factors  were  all 
related  to  the  theme  of  handling  arrivals  and  departures. 
Complexity  factors  appeared  to  influence  controllers  in  moving 
aircraft  away  from  their  original  flight  plans  to  achieve  correct 
spacing,  and  to  reduce  the  potential  for  conflict.  Further 
research  is  recommended  to  validate  these  preliminary  findings  and 
to  extend  the  investigation  of  controller  decision  making. 
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The  purpose  of  this  research  was  to  document  the  cognitive 
foundations  of  air  traffic  control  (ATC)  .  The  focus  was  on  two 
components  of  cognitive  processing  in  ATC:  cognitive  structures 
and  decision-making  strategies.  The  term  cognitive  structure 
refers  to  any  organization  of  ATC-specific  knowledge,  for  example, 
the  controller's  knowledge  about  ATC  procedures,  sector  geography, 
flight  characteristics,  and  previously-experienced  air  traffic 
situations.  Cognitive  structures  are  also  known  as  knowledge 
structures,  or  conceptual  structures.  Because  these  structures 
are  thought  to  guide  decision  making,  they  are  the  key 
underpinnings  for  any  further  investigation  of  controllers' 
cognitive  processes. 

Documentation  of  knowledge  structures  permits  the  investigation  of 
their  influence  on  decision-making  strategies.  A  strategy  is  a 
goal-directed  use  of  resources,  over  time,  in  response  to  a 
situation  that  calls  for  judgment  and  choice  among  options. 
Because  cognitive  structures  and  decision-making  strategies  are  not 
readily  observable,  it  was  necessary  to  devise  a  research  plan  that 
would  produce  valid,  reliable  behavioral  data  on  which  to  base 
inferences  about  internal  processes. 

An  updated  version  of  the  first  report  in  this  series,  "Review  and 
Evaluation  of  Applied  Research  Techniques  for  Documenting  Cognitive 
Processes  in  Air  Traffic  Control"  (Mogford,  Harwood,  Murphy,  and 
Roske-Hof strand,  1992)  ,  reviewed  methods  and  techniques  for 
exploring  cognitive  structures  and  decision  making  in  ATC.  That 
report  also  recommended  appropriate  tools  for  research  in  the  ATC 
environment.  Suitable  techniques  Include  retrospective 
verbalization,  structured  probed  interviews,  video  analysis, 
multidimensional  scaling  (MDS) ,  and  Pathfinder  analysis. 

The  work  described  in  the  present  paper  made  use  of  the  recommended 
methods  to  explore  ATC  knowledge  structures  and  decision-making 
strategies.  Knowledge  structures,  relating  to  the  ATC  environment, 
were  addressed  first,  using  the  assumption  that  they  provide  both 
a  context  and  a  basis  for  decision  making. 

Because  the  en  route  sector  is  the  focus  of  the  en  route  traffic 
situation,  its  features  and  constraints  bound  the  options  available 
to  the  controller.  Given  that  sectors  may  be  alike  in  some  ways, 
yet  widely  different  in  others,  it  is  not  possible  to  describe, 
much  less  predict,  controller  decision-making  performance  unless 
sector  complexity  is  taken  into  account. 

Thus,  the  first  phase  of  the  study  focused  on  an  investigation  of 
en  route  sector  complexity,  and  the  second  phase  took  initial  steps 
toward  investigating  the  effects  of  sector  complexity  on  ATC 
decision  making  in  an  en  route  environment. 


i>i  scoeE. 

Under  CTA,  INCORPORATED • s  (CTA's)  contract  with  the  Federal 
Aviation  Administration  (FAA)  Technical  Center,  this  specific 
research  task  included  two  major  activities: 

a.  Activity  l,  Review  and  evaluation  of  techniques  for 
documenting  cognitive  processes  in  ATC. 

b.  Activity  2,  Measurement  of  structures  and  strategies. 

Phase  I  -  Measurement  of  controllers'  cognitive 
structuring  of  knowledge  about  sector  complexity. 

Phase  II  -  Measurement  of  decision  making  strategies  in 
the  context  of  findings  about  sector  complexity. 

Activity  2  research  followed  activity  1,  which  is  docvimented  by 
Mogford,  et  al.,  (1992).  The  participating  ATC  facility  was  the 
Jacksonville  Air  Route  Traffic  Control  Center  (ARTCC) ,  located  near 
Jacksonville,  Florida. 
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r  OVERVIEW. 


Activity  2  research  begins  with  Phase  I,  cognitive  structures. 
First,  a  preliminary  study  is  described  that  established  the 
context  for  the  research  on  sector  complexity,  and  tested  candidate 
research  techniques.  Some  initial  insight  into  controllers' 
concepts  about  sector  complexity  emerged. 

An  account  of  the  main  study  follows  with  a  more  extensive 
exploration  of  controller  knowledge  about  en  route  sectors.  These 
efforts  resulted  in  an  initial  list  of  sector  complexity  factors. 
The  final  stage  of  Phase  I  was  to  attempt  to  determine  which  of 
these  factors  was  most  related  to  overall  sector  complexity,  as 
judged  by  a  group  of  Traffic  Management  Unit  (TMU)  controllers. 


Phase  II  research  sought  to  establish  a  relationship  between  the 
findings  regarding  sector  complexity,  and  decision  making.  A 
preliminary  study  was  completed  that  examined  a  number  of  aspects 
of  controller  behavior,  and  made  a  start  at  linking  decisions  and 
actions  to  sector  complexity.  Recommendations  are  made  for  further 
research. 


2.  PHASE  I  -  CONCEPTUAL  STRUCTURES  OF  SECTOR  COMPLEXITY. 

2.1  BACKGROUND  AND  APPROACH. 

When  performing  ATC  tasks,  a  controller's  area  of  concern  is  a 
well-defined  volume  of  airspace  called  a  sector.  Each  terminal 
area,  or  en  route  center,  includes  many  contiguous  sectors.  A 
sample  sector  map  from  the  Jacksonville  ARTCC  is  shown  in  figure  1. 


2 


SECTOR  34 

SEMiNOUE-  HI 
maAo-mn 

VIDEO  MAP  i:HW' 
s^RADAR  SITE  -QHN 


\ 


^,**<«**«i«***ff‘S'»”**\  \  \ 


•n.<M  »  M«i 


MOODY  3 
MOA/ATCAA 

\  •  *itr« 


o  \ 

IM— MP  \ 


<  cewevA  •  HI 


o~»» 

I^IM/  •  mr) 


- 


>  V  »*«■» 

\  \ 

\  \,  < 


— mUM  “n-T — -••* 


■APKIL  4.  1991 


Figure  1.  Sector  Map  for  the  Seminole  Sector, 
Jacksonville,  Florida  ARTCC 


Seminole  sector  is  one  of  38  sectors  in  the  Center  through  which 
aircraft  fly  on  their  way  to  destinations  in  the  United  States  and 
abroad. 

The  characteristics  of  a  sector,  in  terms  of  airways,  navigational 
aids,  airports,  traffic  flow,  weather,  and  other  variables,  define 
the  controller's  tasks  and  decisions.  Examining  the  ways  in  which 
controllers  structure  their  knowledge  about  sector  complexity  can 
establish  the  basis  for  further  research  on  controller  planning, 
problem  solving,  and  decision  making.  The  purpose  of  any  such 
investigation  is  to  disclose  data  and  information  about  controller 
cognition  that  might  help  in  improving  ATC  efficiency  and 
effectiveness,  e.g.,  through  improvements  in  the  design  of  sectors, 
planning  and  decision  aids,  information  displays,  and  controller 
training. 

As  contributors  to  task  demand,  sector  characteristics  can  be 
thought  of  as  workload  generators.  Workload  is  the  controller's 
subjective  response  to  the  "objective"  conditions  which  create 
sector  complexity.  Stein  (1985)  found  that  global  controller 
workload  could  be  predicted  by  such  factors  as 
clustering/complexity  of  traffic  patterns,  frequency  of  handoffs 
outbound,  number  of  flights  handled,  and  frequency  of  handoffs 
inbound.  However,  the  variables  used  by  Stein  did  not  focus  on  the 
direct  contribution  made  to  workload  by  sector  characteristics  (as 
opposed  to  traffic  characteristics) . 

Hurst  and  Rose  (1978)  conducted  a  study  to  determine  the  principal 
behavioral  stressors  in  the  ATC  environment.  Four  expert  observers 
rated  the  level  of  degree  of  activity  and  behavioral  arousal  of 
controllers  working  47  radar  sectors  in  the  Boston  and  New  York 
areas.  It  was  found  that  the  activity  and  arousal  measures  were 
significantly  related  to  peak  traffic  count  and  duration  of  radio 
communications.  A  multiple  regression  analysis  showed  that  peak 
traffic  alone  accounted  for  53  percent  of  the  variance  in  activity 
and  arousal.  No  other  variable  moderated  this  effect,  including 
sector-related  factors  such  as  sector  type  and  size.  The  authors 
stated  that  peak  traffic  may  be  the  most  potent  and  generalizable 
contributor  to  controller  workload. 

The  FAA  currently  uses  a  formula  to  evaluate  the  complexity  of  en 
route  airspace  on  a  yearly  basis.  The  procedure  is  described  in 
FAA  Order  7210.46,  "Establishment  and  Validation  of  En  Route 
Sectors."  The  formula  consists  of  counts  of  the  following 
activities,  for  a  selected  time  period,  at  each  en  route  center: 

a .  Departures . 

b.  Arrivals. 
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c.  Radar  vectored  arrivals. 

d.  En  route  (requiring  control  functions) . 

e.  En  route  (no  control  functions) . 

f .  Emergencies . 

g.  Special  flights. 

h.  Coordination  (additional  points  when  above  functions 
require  coordination) . 

There  is  some  concern  within  the  FAA,  however,  that  this  formula 
does  not  represent  all  of  the  important  aspects  of  airspace 
complexity.  In  order  to  address  this  matter,  Grossberg  (1989) 
conducted  research  on  sector  complexity  at  the  Chicago  ARTCC.  He 
defined  complexity  as  "A  construct,  referring  to  the 
characteristics,  dynamic  and  static,  affecting  the  rate  at  which 
workload  increases,”  and  set  out  to  define  qualitative  guidelines 
for  limiting  operational  complexity.  He  noted  that  the  PAA's 
standard  sector  complexity  formula  did  not  adequately  account  for 
the  number  of  operational  ATC  errors  occurring  in  the  field. 

Based  on  an  earlier  FAA  study  (Robertson,  Grossberg,  and  Richards, 
1979) ,  Grossberg  (1989)  defined  12  candidate  complexity  factors: 

a.  Large  sector  airspace. 

b.  Small  sector  airspace. 

c.  Brief  sector  flight  time. 

d.  Quick  procedural  adjustments. 

e.  Frequent  coordination. 

f.  Complex  control  adjustments. 

g.  Complex  flight  paths. 

h.  Intersecting  flight  paths. 

i.  Climbing/ descending  flights. 

j.  Mix  of  aircraft  types. 

k.  Frequency  of  holding/path  stretching. 

l.  Heavy  traffic. 

Grossberg  (1989)  asked  97  ATC  specialists  and  supervisors  to  rate 
each  of  the  Chicago  ARTCC  sectors  on  the  degree  to  which  each 
factor  increased  sector  difficulty  or  complexity.  The  factors 
cited  most  frequently  were:  control  adjustments,  such  as  merging, 
spacing,  and  speed  changes;  climbing  and  descending  flight  paths; 
and  mix  of  aircraft  types. 

Finding  that  the  factors  could  distinguish  between  sectors, 
Grossberg  (1989)  derived  a  "sector  complexity  index"  by  tedcing  the 
most  reliable,  non-traffic  related  factors  and  calculating  the  mean 
of  the  highest  four  ratings  for  any  given  sector.  For  27  of  the 


Chicago  ARTCC  sectors,  he  found  that  the  resulting  sector 
complexity  ratings  were  highly  correlated  with  the  number  of 
operational  errors;  as  ratings  increased,  so  did  operational 
errors . 

Grossberg's  work  on  identifying  salient  complexity  factors  showed 
that  the  existing  rating  system  was  lacking  in  detail,  and  that 
improvements  were  possible.  The  work  undertaken  in  Activity  2, 
Phase  I,  continued  the  research  on  sector  complexity  in  order  to 
identify  additional  factors,  extend  Grossberg's  findings,  test  the 
selected  knowledge  exploration  methods,  and  establish  a  basis  for 
the  study  of  controller  decision  making. 

In  Phase  I,  the  previously  identified  knowledge  exploration 
techniques  (Mogford,  et  al.,  1992)  were  classified  into  two  general 
types,  direct  and  indirect.  Methods  that  rely  on  verbal  reports, 
interviews,  or  simple  questionnaires  were  considered  direct 
approaches,  and  methods  that  involve  a  statistical  analysis  of 
controller  judgments  (including  MDS  and  Pathfinder)  were  termed 
indirect  approaches. 

It  is  obvious  why  direct  approaches  would  be  part  of  an^  study  that 
seeks  to  uncover  controller  knowledge  about  airspace 
characteristics.  However,  more  needs  to  be  said  about  the  indirect 
approaches . 

When  experts  are  consulted  to  describe  what  they  know  about  a 
particular  domain,  researchers  are  often  concerned  that  a 
significant  amount  of  expert  knowledge  is  so  well  ingrained  that  it 
is  not  readily  accessible  for  discussion.  Experts  automatically 
know  how  to  accomplish  a  task,  but  may  have  difficulty  describing 
all  of  the  steps  in  the  process,  information  used,  decisions  made, 
etc.  Given  this  problem,  an  alternative  is  to  observe  or  measure 
task-related  behaviors  that  can  provide  insight  into  underlying 
knowledge  structures.  For  this  reason,  indirect  approaches,  such 
as  MDS  and  Pathfinder,  have  been  developed. 

MDS  and  Pathfinder  are  based  on  the  assumption  that  if  one  wants  to 
find  out  about  the  relationships  between  concepts  or  objects,  a 
useful  approach  is  to  ask  for  similarity  comparisons  between  them. 
Without  preconceptions  or  external  guidance,  subject  matter  experts 
(SMEs)  are  asked  to  rate  the  similarity  between  pairs  of  concepts. 
Perhaps  without  fully  being  aware  of  it,  they  will  use  a  set  of 
criteria  to  accomplish  this.  In  the  case  of  en  route  sectors,  they 
may  compare  them  for  traffic  load,  size  of  airspace,  number  of 
military  flights,  and  other  factors.  Overall  sector  similarity 
will  be  judged  on  how  close  a  given  pair  of  sectors  are  on  these 
dimensions. 
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similarity  judgments  are  also  called  distance  judgments  or 
proximity  judgments,  because  they  are  estimates  of  how  close  (or 
how  far  apart)  the  objects  being  compared  are  on  the  concept  of 
interest  (in  this  case,  complexity).  MDS  is  useful  because  it  can 
take  similarity  judgments  between  pairs  of  items  as  raw  input  and 
display  a  map,  which  spells  out  the  relationships  between  the 
objects.  It  is  then  possible,  by  analyzing  the  pattern,  to 
determine  what  characteristics  or  dimensions  are  being  used  by  the 
SME  in  making  the  original  comparisons. 

For  en  route  controllers,  MDS  can  help  uncover  the  complexity 
factors  that  are  considered  important  when  comparing  different 
sectors.  Pathfinder  produces  similar  results,  but  shows  more 
specific  information  about  the  linkages  between  concepts.  (See 
appendix  A  for  additional  background  information  on  MDS  and 
Pathfinder. ) 

4 

In  Phase  I,  direct  techniques  for  knowledge  exploration  included 
SME  rankings  of  sectors  by  complexity,  descriptions  of  complexity 
factors,  ratings  of  sectors  on  complexity  factors,  and  ratings  of 
overall  sector  complexity.  Indirect  methods  relied  on  collecting 
SME  similarity  judgments  based  on  comparing  sectors.  In  this  case 
the  researchers  directed  controllers  to  compare  sectors 
specifically  on  their  complexity.  MDS  could  then  be  expected  to 
help  illustrate  the  sector  characteristics  that  were  being  used  by 
controllers  to  make  the  comparisons. 

Phase  I  of  the  research,  described  in  the  current  report,  involved 
the  application  of  both  direct  and  indirect  methods  to  derive  and 
prioritize  a  set  of  airspace  complexity  factors.  The  work  took 
place  in  two  stages,  item  development,  and  factor  selection.  Item 
development  resulted  in  a  list  of  candidate  sector  complexity 
factors,  and  employed  both  direct  and  indirect  approaches. 

Once  this  had  been  accomplished,  the  list  of  factors  was  used  to 
evaluate  all  of  the  sectors  in  the  Center,  and  the  resulting 
complexity  ratings  were  analyzed  for  their  ability  to  account  for 
overall  sector  complexity.  An  additional  goal  was  to  determine  the 

*  usefulness  of  each  approach  (direct  and  indirect)  in  generating 
salient  factors  (Mogford,  Murphy,  and  Guttman,  1993) . 

*  2.2  PRELIMINARY  STUDY. 

The  preliminary  study  of  cognitive  structures  was  intended  to  test 
the  methodologies  planned  for  the  main  study  on  controller 
knowledge  about  sector  complexity.  Two  visits  were  made  to  the 
Jacksonville  ARTCC  to  develop  and  evaluate  the  approach  for  the 
main  study. 


7 


The  Jacksonville  Airspace  Procedures  and  Training  Office  personnel 
were  extremely  helpful  in  orienting  the  CTA  research  team  to  the 
Center's  airspace  and  procedures.  Six  Airspace  and  Procedures 
Specialists  (APSs)  and  foxur  Full  Performance  Level  (FPL) 
controllers  from  the  West  Specialization  Area  were  recruited  for 
the  preliminary  study.  All  six  of  the  Center's  APSs  participated. 
FPL  controller  selection  was  not  random,  but  was  based  on 
availability  and  scheduling  requirements. 

2.2.2  Materials  and  Apparatus. 

Two  data  collection  methods  were  developed  and  tested.  These  were 
structured  Interview  techniques,  and  paired  comparisons  of  airspace 
maps  (which  created  data  for  MDS  analysis) .  The  West 
Specialization  Area  was  used  for  the  preliminary  study.  It  is 
composed  of  seven  sectors:  Crestview,  Way cross,  Albany,  Ashburn, 
Brewton,  Tallahassee,  and  Nepta.  (Although  controllers  use 
numerical  identifiers  for  their  sectors,  the  research  team  found  it 
easier  to  employ  the  sector  names.) 

Response  forms  were  designed  to  record  data  gained  from  structured 
Interviews.  An  audio  tape  recorder  was  employed  to  captiire 
^narratives  for  later  analysis. 

Airspace  maps  were  collected  for  all  of  the  Jacksonville  ARTCC 
sectors.  In  addition,  photographs  (showing  traffic)  were  taken  of 
the  radar  maps  of  all  sectors.  The  maps,  and  enlargements  of  the 
photographs  for  the  West  Specialization  Area,  were  attached  to 
stiff  cardboard  backings,  and  placed  on  stands  to  serve  as  stimuli 
for  the  paired-comparison  task. 

FAA  sector  density  and  complexity  ratings  (from  the  most  recent 
Jacksonville  ARTCC  Annual  Center  Review)  were  also  collected. 


2.2.3  Procedure. 

Working  in  a  group,  but  without  discussion,  the  six  APSs  rated  all 
38  sectors  in  the  Center  for  their  own  level  of  familiarity  with 
each  sector,  the  overall  complexity  of  the  sector,  and  the  normal 
difficulty  of  controlling  aircraft  in  the  sector.  They  also  listed 
important  sector  complexity  factors.  (Sector  complexity  was  not 
defined  as  a  term,  but  its  meaning  was  left  up  to  the  judgment  of 
the  participant.  The  definition  was  left  purposely  open-ended  so 
that  all  possible  contributing  factors  could  be  identified.)  All 
responses  were  recorded  on  forms  designed  for  each  task. 


Each  FPL  was  then  interviewed  separately  and  made  responses  on  the 
appropriate  forms.  Participants  ranked  sectors  on  complexity,  by 
sorting  either  photographs  or  maps  of  sectors  into  order  of 
complexity.  Each  participant  was  given  either  photographs  or  maps 
(order  was  reversed  for  each  subsequent  participant) ,  and  asked  to 
arrange  them  into  a  stack  with  the  most  complex  sector  uppermost. 
After  completing  this  sorting  task,  the  participant  described  the 
reasons  for  the  rankings.  This  debriefing  was  tape  recorded. 
Participants  were  then  asked  to  list  important  complexity  factors. 

Finally,  each  FPL  made  judgments  of  the  similarity  in  complexity 
between  all  possible  pairs  of  sectors  in  the  West  Area  using  both 
photographs  and  maps.  Either  photographs  or  maps  were  used  first 
(corresponding  to  the  stimulus  type  used  in  the  palrticipant 's  first 
judgment  task) .  Each  possible  pair  of  stimuli  was  placed  on 
picture  stands  (small  easels)  in  front  of  the  participant,  who 
recorded  judgments  of  the  two  sectors*  similarity  in  complexity  on 
a  special  form. 

2.3  RESULTS. 

CaffiilUrityi  Coffplsxitv;*.  and. 

The  six  APSs  had  been  asked  to  rate  all  of  the  38  Jacksonville 
ARTCC  sectors  for  familiarity,  complexity,  and  difficulty.  It  was 
discovered  during  data  collection  that  each  APS  was  familiar  with 
a  different  set  of  sectors.  Whe<i  it  came  time  to  rate  the  38 
sectors  for  complexity  and  difficulty,  it  was  not  possible  to 
collect  a  full  set  of  judgments  from  each  APS.  Their  lack  of 
familiarity  with  some  sectors  resulted  in  several  missing  ratings 
in  each  case.  The  missing  data  were  for  different  sectors  for  each 
APS.  This  made  it  impossible  to  correlate  complexity  and 
difficulty  judgments  for  all  sectors  between  the  individual  APSs. 

It  was  possible,  however,  to  average  and  correlate  the  available 
complexity  and  difficulty  ratings  across  participants,  and  generate 
a  complexity  and  difficulty  rating  for  each  sector.  The  product- 
moment  correlation  between  ratings  of  complexity,  and  difficulty  of 
the  sectors  was  r  »  .78,  p<.01.^  This  indicates  that  there  was  a 
high  degree  of  relationship  between  the  perceived  complexity  of  a 
sector,  and  the  judgment  of  how  difficult  it  is  to  work. 


1  The  Pearson  Product-Moment  Correlation  Coefficient  (r)  ie  a  statistical  method 
of  detenninin9  the  degree  of  relationship  between  two  sets  of  data  (such  as 
ratings  of  complexity  and  difficulty)  about  the  same  objects.  The  range  of  r  is 
between  zero  (no  relationship)  and  1.0  (high  relationship).  The  "p"  value 
associated  with  the  correlation  indicates  the  probability  that  the  correlation 
is  the  result  of  chance  factors  (in  this  case  1/100).  The  lower  the  p  value,  the 
higher  the  probability  that  the  relationship  is  dependable  and  meaningful. 
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It  was  important,  for  the  purpose  of  assessing  complexity  factors 
applying  to  all  sectors,  that  a  group  of  Jacksonville  ARTCC 
personnel  be  familiar  with  a  large  proportion  of  the  sectors  in  the 
Center's  airspace.  Given  that  the  APSs  had  indicated  feuailiarity 
with  only  some  of  the  38  sectors,  they  were  asked  to  become 
acquainted  with  as  many  of  their  less  familiar  sectors  as  possible 
in  preparation  for  the  main  study. 

To  supplement  these  APS  findings,  the  knowledge  of  the  TMU  staff 
was  also  surveyed,  with  the  expectation  that  some  of  them  might  be 
familiar  with  more  of  the  Center's  sectors.  Of  the  11  TMU 
personnel  who  responded,  4  claimed  an  average  or  higher  familiarity 
with  most  of  the  sectors  in  the  Center. 

2.3.2 _ Sector  Sorts. 

During  data  collection,  each  of  the  four  FPLs  from  the  West  Area 
ranked  the  seven  sectors  in  this  area  by  complexity.  The  Kendall 
correlation^  between  the  four  controllers  was  x^(6,  N  ■>  4)  »  21.42, 
p  S.0015.  Crestview  was  consistently  chosen  as  the  most  complex 
sector.  These  results  indicate  a  high  degree  of  agreement  between 
the  controllers  in  the  preliminary  study,  regarding  the  level  of 
complexity  of  sectors  within  the  West  Specialization  Area.  They  do 
not,  however,  explain  why  Crestview  was  considered  to  be  the  most 
complex  sector. 

2f3»3 _ ggBPlexitY  fcjgtg. 

As  part  of  the  test  procedure,  the  six  APSs,  and  four  FPLs  from  the 
West  Area,  had  been  asked  to  list  the  factors  they  thought 
contributed  to  sector  complexity.  A  total  of  61  factors  were 
collected.  Discussions  with  the  Jacksonville  ARTCC  training  office 
permitted  removal  of  redxmdant  factors,  and  resulted  in  a  list  of 
19  complexity  factors.  It  was  possible  to  rank  most  of  the  factors 
by  coiinting  how  many  times  each  was  mentioned.  Table  1  shows  the 
factors  and  the  number  of  votes  each  received. 

The  factors  in  table  1  were  then  combined  with  the  12  complexity 
factors  defined  in  previous  research  on  sector  complexity  by 
Grossberg  (1989) .  Fxirther  redundancies  were  removed,  resulting  in 
a  list  of  24  Initial  Factors.  These  were  later  evaluated  for 
Importance  by  the  participants  involved  in  the  main  study,  and  used 
to  gather  descriptive  information  on  the  Jacksonville  ARTCC  sectors 
to  assist  with  subsequent  MDS  analyses.  The  list  of  the  24  factors 
is  found  in  table  2. 


2  The  Kendall  correlation  coefficient  is  another  measure  of  the  degree  of 
relationship,  but  is  used  to  measure  agreement  between  judges  who  have  been  aslced 
to  rank  a  set  of  people  or  objects.  It  ranges  from  zero  (no  agreement)  to  1.0 
(complete  agreement).  Unlike  the  Pearson  correlation,  it  offers  the  ability  to 
cooipute  one  measure  of  agreement  between  several  judges. 
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RESTRICTED  AREAS,  WARNING  AREAS,  MOAS,  AND 
NOTIFICATION  TIME 


FACTOR  MAMS 


WEATHER 


AIRCRAFT  MIX  (COMMERCIAL,  MILITARY,  PRIVATE 


CQNVERGING/CRQSSING  TRAFFIC  _ 


SECTOR  BOUNDARIES /SECTOR  SIZE 


CLIMBING/DESCENDING  TRAFFIC  HEAD-ON 


VOLUME  OF  TRAFFIC 


COORDINATION 


PERSONALITY  OF  FELLOW  CONTROLLERS 


LACK  OF  RADAR  AND  RADIO  COVERAGE 


MULTIPLE  FUNCTIONS  APPROACH  CONTROL,  TERMINAL 
FEEDER,  EN  ROUTE  IN  TRAIL  SPACING,  ETC. 


NUMBER  OF  FACILITIES  TO  DEAL  WITH  (SECTORS, 
CENTERS,  APPROACH  CONTROL,  MILITARY) 


DEPARTURE  AND  ARRIVAL  TRANSITIONS  (STARS/SIDS 


DEPARTING  AND  ARRIVING  AIRCRAFT 


MILITARY  REQUIREMENTS  CONTRARY  TO  NORMAL  FLOWS 


AIRLINE  HUBBING 


IRREGULAR  ALTITUDE  STRATIFICATION 


EQUIPMENT 


FREQUENCY  CONGESTION 


OCCDRRBMCE 
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Tabltt  2.  Initial  Complexity  Factors  as  Suggested  by  APS 
Staff  and  West  Area  FPLs  in  the  Preliminary  Study. 
(Factors  are  not  listed  in  order  of  importance) . 


!■ 

NUMBER  OF  RESTRICTED  AREAS,  WARNING  AREAS,  MOAS,  AND 
NOTIFICATION  TIME 

2. 

FREOUENCY  OF  PROBLEM  WEATHER 

3. 

MIX  OF  AIRCRAFT  TYPES  (COMMERCIAL.  MILITARY.  PRIVATE) 

4. 

NUMBER  OF  INTERSECTING  FLIGHT  PATHS 

5. 

PROBLEMS  WITH  SECTOR  BOUNDARIES 

6. 

SIZE  OF  SECTOR  AIRSPACE 

7. 

NUMBER  OF  CLIMBING/DESCENDING  FLIGHTS 

8. 

AMOUNT  OF  HEAVY  TRAFFIC 

9. 

FREOUENT  COORDINATION 

ADEQUACY  OF  RADAR  (AND  RADIO)  COVERAGE 

11. 

MULTIPLE  FUNCTIONS  (APPROACH  CONTROL,  TERMINAL  FEEDER, 
EN  ROUTE  IN  TRAIL  SPACING.  ETC.) 

12. 

NUMBER  OF  FACILITIES  TO  DEAL  WITH  (SECTORS,  CENTERS, 
APPROACH  CONTROL.  MILITARY) 

13. 

NUMBER  OF  DEPARTURE  AND  ARRIVAL  TRANSITIONS 
(STARS/SIDS) 

14. 

NUMBER  OF  DEPARTING  AND  ARRIVING  AIRCRAFT 

15. 

FREQUENCY  OF  MILITARY  REQUIREMENTS  CONTRARY  TO  NORMAL 
FLOWS 

AMOUNT  OF  AIRLINE  RUBBING 

warn 

IRREGULAR  ALTITUDE  STRATIFICATION 

Kn 

ADEQUACY  OF  EQUIPMENT 

WEm 

AMOUNT  OF  RADIO  FREOUENCY  CONGESTION 

BRIEF  SECTOR  FLIGHT  TIME 

Esmm 

FREQUENCY  OF  PROCEDURAL  ADJUSTMENTS 

22. 

FREQUENCY  OF  COMPLEX  CONTROL  ADJUSTMENTS  (MERGING, 
SPACING.  SPEED.  . . . ) 

BOH 

COMPLEX  FLIGHT  PATHS 

EM 

FREQUENT  HOLD /PATH  STRETCHING 

2,3,4  Complexity  Similarity  Ratings. 

2, 3. 4.1  Correlations  between  Judgments. 

During  data  collection,  the  four  FPLs  had  made  complexity 
similarity  ratings  between  the  seven  sectors  in  the  West 
Specialization  Area.  (This  was  for  the  purpose  of  testing  the  MOS 
approach  to  developing  complexity  factors.)  When  Kendall 
correlations  were  calculated  between  the  judgments  of  the  four  West 
Area  FPLs  for  photographs  and  maps,  it  appeared  that  the  group  was 
generally  in  agreement  about  the  similarity  in  complexity  of  the 
sectors.  The  result  for  photographs  was  x^(20,  N  «  4)  »=  43.78,  p 
“  .0016,  and  for  maps  it  was  x^(20,  N  =  4)  =  43.20,  p  *  .0019. 

The  researchers  were  interested  in  the  differences  between  using 
radar  screen  photographs,  as  opposed  to  sector  maps,  for  collecting 
judgments  about  sectors.  When  product ^moment  correlations  between 
each  individual's  complexity  similarity  judgments,  using 
photographs  and  maps,  were  computed,  results  ranged  from  0.78  to 
0.94  (all  significant  at  p  <  .01).  The  Kendall  correlation  between 
all  judgments  of  photographs  and  maps  was  x^(20,  N  »  8)  »  85.01,  p 
»  .0000.  These  results  show  that  there  was  not  much  difference  in 
complexity  similarity  ratings  when  participants  based  their  ratings 
on  photographs  with  traffic,  as  opposed  to  sector  maps  without 
traffic. 

When  asked  for  an  opinion  on  which  stimuli  to  use  in  the  main 
study,  the  FPLs  said  they  preferred  screen  photographs,  while  the 
APSs  selected  sector  maps.  Each  group  chose  the  medium  they  worked 
with  most  of  the  time.  Since  results  for  photographs  and  maps  were 
virtually  interchangeable,  use  of  either  photos  or  maps  would  have 
been  valid.  Due  to  technical  difficulties  in  collecting  a  complete 
set  of  radar  screen  photographs,  sector  maps  were  chosen  for 
subsequent  data  collection  efforts. 


2. 3. 4. 2  MDS. 

To  further  explore  the  data  from  the  preliminary  study,  an  MDS 
analysis  was  conducted  on  the  sector  complexity  similarity  data 
sets  collected,  using  photographs  and  maps.  The  purpose  of  this 
analysis  was  to  examine  the  effectiveness  of  MDS  as  an  indirect 
knowledge-exploration  tool. 

The  proximity  data  (i.e.,  the  ratings  of  paired  sectors  for 
similarity  in  complexity)  were  analyzed,  using  an  MDS  procedure 
known  as  INDSCAL,  a  subroutine  of  alternating  least  squares  scaling 
(ALSCAL)  in  the  Statistical  Package  for  the  Social  Sciences  (SPSS) . 
INDSCAL  combines  sets  of  proximity  measures  for  a  group  of 
participants,  and  creates  a  shared  stimulus  space.  Using  this 
space,  it  is  possible  to  analyze  the  relationships  between  a  set  of 
objects,  such  as  sectors. 


13 


INDSCAL  computes  how  efficiently  it  has  accounted  for  each 
pairticipant's  proximity  data  in  the  shared  space,  and  also 
calculates  a  weight  space,  which  shows  to  what  degree  each 
participant  emphasizes  the  common  MDS  dimensions.  The  goodness  of 
fit  of  the  solution  or  effectiveness  is  measured  by  minimizing 
stress^  and  maximizing  the  amount  of  variance  (or  information) 
accounted  for 

Similar  two-dimensional  MDS  stimulus  spaces  resulted  for  photos  and 
maps,  and  only  the  maps  data  were  analyzed  further.  The  result  for 
the  maps  data  (shown  in  figure  2)  achieved  a  stress  of  .13  and 
accounted  for  91  percent  of  the  variance  in  the  data.  The  MDS 
space  for  the  maps  was  interpreted  by  the  experimenters  using 
sector  information  obtained'  at  Jacksonville's  ARTCC  with  the 
assistance  of  the  Center's  training  staff.  One  dimension  of  the 
MDS  space  was  related  to  sector  traffic  volume  and  the  amount  of 
military  traffic,  while  the  second  dimension  was  concerned  with 
traffic  movement  (such  as  altitude,  overflights  vs.  departures/ 
arrivals,  and  number  of  procedures) .  For  the  four  West  Area  FPLs, 
these  appeared  to  be  the  underlying  factors  guiding  their 
complexity  comparisons  of  the  seven  sectors. 

Each  participant's  complexity-similarity  judgments  were  well 
accounted  for  in  the  combined  MDS  space,  but  there  was  an 
indication  that  they  each  used  the  underlying  complexity  dimensions 
differently  when  making  their  judgments.  Figure  3  shows  the  weight 
space  for  the  four  FPLs,  and  indicates  how  much  each  participant 
emphasized  each  dimension  when  judging  the  similarity  of  complexity 
between  pairs  of  sectors.^ 

Pathfinder  analysis  was  considered  as  another  way  to  explore  the 
structure  of  the  sector  complexity  information.  The  Pathfinder 
algorithm  generates  a  family  of  link-weighted  networks  from  a  set 
of  distance  data  (see  appendix  A) .  A  derived  Pathfinder  network 
consists  of  a  set  of  concepts  and  links  that  directly  connect  pairs 
of  highly-related  concepts.  This  network  represents  the  basic 
organization  of  existing  knowledge  structures  or  elements  based  on 
an  individual's  experience. 


3  Stress  is  the  square  root  of  a  normalized  residual  sum  of  squares  and  should 
be  below  3.0  in  a  well-fitting  MDS  solution. 

4  ranges  from  zero  to  100  percent.  MDS  results  that  produce  an  R^  of  over  75 
percent  are  considered  acceptable. 

5  If  subjects  used  each  dimension  about  equally,  they  would  lie  on  a  diagonal 
passing  through  the  zero  point  on  the  %raight  space  graph.  Deviation  frtxa  the 
diagonal  indicates  a  preference  for  one  dimension  or  the  other.  Distance  from 
zero  on  either  dimension  indicates  the  degree  of  emphasis  on  that  dimension. 
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DERIVED  STIMULUS  CC»JFIGURATIC»i: 

DIMEKSIC»I  1  (HORIZONTAL)  VS  DIMENSION  2  (VERTICJIL) 
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Figure  2.  INDSCAL  Space  for  the  Four  West  Area  FPLs 
in  the  Preliminary  Study 
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Figure  3.  Weight  Space  for  the  Four  West  Area  FPLs 
in  the  Preliminary  Study 
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Pathfinder  analysis  was  applied  to  the  averaged  similarity  judgment 
matrices  for  maps  rated  by  the  West  Area  controllers  in  the 
preliminary  study,  as  shown  in  figure  4.  The  complexity  factors 
were  identified  that  appeared  to  be  the  basis  for  links  between 
sectors.  These  factors  proved  to  be  similar  to  those  derived  from 
the  MDS  results.  Based  on  this  finding,  and  the  large  amount  of 
time  required  to  interpret  each  Pathfinder  graph,  it  was  decided 
that  Pathfinder  analysis  of  the  sector  proximity  data  would 
probably  not  offer  significantly  more  information  than  MDS. 
Therefore,  Pathfinder  was  reserved  for  the  analysis  of  the 
complexity  factors  identified  through  the  direct  and  indirect 
methods.  This  Pathfinder  analysis  is  discussed  in  section  2.9.3. 

2.4  MAIN  STUDY. 

Methods  and  materials  used  in  the  main  study  were  selected  on  the 
basis  of  outcomes  of  the  preliminary  study.  To  develop  a  list  of 
Candidate  Factors,  the  CTA  research  team  made  a  third  visit  to  the 
Jacksonville  ARTCC.  Both  direct  (questionnaire)  and  indirect 
(similarity  judgments)  approaches  were  used  to  collect  data  on 
sector  complexity  on  a  larger  scale. 

_ Participants. 

A  total  of  30  controllers  (25  FPLs  and  5  trainees  or 
Developments  Is)  were  involved  in  the  main  study;  5  from  each 
specialization  area.  Other  participants  included  5  controller 
trainees,  3  TMU  staff,  and  3  APSs.  The  41  participants  were  all 
volunteers,  and  were  selected  on  the  basis  of  availability  during 
the  time  period  of  the  data  collection  visit. 

Each  FPL's  primary  feuniliarity  was  with  the  seven  to  nine  sectors 
within  a  specialization  area.  Some  specialization  areas  were 
composed  of  sectors  of  mainly  one  type,  such  as  high  or  low 
altitude.  Airspace  Procedures  and  TMU  participants  were  selected 
on  the  basis  of  their  feuniliarity  with  the  sectors  in  the  Center 
(as  previously  tested  by  questionnaire) ,  as  well  as  their 
availability  on  the  days  planned  for  the  study. 

2.4.2  Materials  and  Apparatus. 

Tools  for  data  collection  included  both  computerized  and  paper- 
based  methods.  A  Macintosh  HyperCard  program  (called  the  "Judgment 
Recorder")  was  developed  for  the  collection  of  complexity 
comparison  judgments.  Given  that  the  participants  in  the 
preliminary  study  made  similar  responses  to  both  maps  and 
photographs  of  Jacksonville  sectors,  it  was  decided  to  use  maps  as 
stimuli,  since  they  were  readily  available.  Sector  maps  were 
scanned  and  digitized  using  a  Hewlett-Packard  scanner.  They  were 
then  converted  to  Macintosh  "PICT"  format,  and  incorporated  into 
the  Judgment  Recorder. 
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The  stimuli  were  presented  on  21-inch,  high-resolution  color 
monitors  connected  to  a  Macintosh  Ilci  or  lies  system. 
Participants  made  responses  by  moving  a  mouse-controlled  pointer, 
which  controlled  stimulus  presentation,  and  selection  of  complexity 
similarity  values  on  an  on-screen  scale. 

The  study  also  employed  several  paper  questionnaires,  including  the 
following  forms  (not  all  forms  were  used  with  all  participants) : 

a.  Request  for  a  listing  of  complexity  factors. 

b.  Order  of  sorting  sectors  for  complexity. 

c.  Ratings  of  sectors  on  24  previously  compiled  complexity 
factors  (derived  from  the  preliminary  study  and  previous  work  by 
Grossberg  (1989) ) . 

d.  Importance  ratings  for  24  complexity  factors. 

e.  Word-Shape  Sorting  Test  (WSST)  (a  test  of  cognitive 
style) . 

2.4.3  Procedure. 

The  computerized  Judgment  Recorder  was  used  to  present  sector  maps 
to  25  FPLs,  5  trainees,  3  APSs,  and  3  TMU  staff  members.  Upon 
arriving  at  the  room  used  for  the  main  study,  participants  filled 
out  a  form  with  their  name,  age,  sex,  and  years  of  experience. 
They  were  then  introduced  to  the  computerized  Judgment  Recorder  and 
completed  a  tutorial.  Following  this,  the  FPLs  from  the  North, 
South,  East,  West,  or  Central  Specialization  Areas,  and 
Oevelopmentals  from  the  East  Area,  first  rated  their  familiarity 
with  the  seven  to  nine  sectors  in  each  area .  They  then  made 
complexity  similarity  comparisons  between  all  possible  pairs  of 
sectors . 

The  Airspace  Procedxires  and  TMU  personnel  completed  the  same 
procedure,  but  did  not  compare  all  possible  pairs.  After  a 
participant  had  made  a  rating  of  familiarity  with  each  sector,  it 
was  possible  to  exclude  all  sectors  that  were  not  well  known.  For 
each  participant,  the  experimenters  evaluated  the  proportion  of  the 
total  number  of  sectors  that  fell  within  each  level  of  familiarity 
on  a  seven-point  scale.  Each  participant  employed  the  scale 
somewhat  differently,  and  a  cutoff  was  selected  in  each  case,  which 
resulted  in  about  10  percent  of  the  least  familiar  sectors  being 
excluded  from  the  paired  comparisons.  As  a  result,  the  Airspace 
Procedures  and  TMU  participants  rated  from  24  to  36  of  the  38 
sectors.  Although  this  required  hundreds  of  paired  comparisons, 
participants  were  encouraged  to  respond  rapidly  without  excessive 
deliberation,  and  an  entire  comparison  session  required  no  more 
than  90  minutes. 
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Sector  maps  were  presented  without  air  traffic.  It  was  assumed 
that,  although  traffic  load,  aircraft  routes,  and  aircraft 
characteristics  might  play  a  part  in  complexity,  having  traffic  on 
the  sector  maps  would  be  misleading  and  possibly  confounding.  The 
presence  of  specific  aircraft  targets  could  have  encouraged 
participants  to  compare  sectors  based  on  the  number  of  aircraft, 
and  the  existence  of,  or  potential  for,  conflicts.  It  also  would 
have  been  nearly  impossible  to  select  representative  and  comparable 
samples  of  traffic  and  ATC  situations  for  each  sector. 

All  participants  were  asked  to  list  the  factors  that  they  thought 
contributed  to  sector  complexity,  and  to  complete  the  HSST.  FPLs 
and  Developmentals  also  sorted  sector  maps  within  their 
specialization  areas  by  level  of  complexity,  and  described  how  they 
determined  the  order.  They  also  completed  two  rating  forms.  These 
forms  employed  an  Initial  set  of  24  complexity  factors  compiled 
from  the  preliminary  study  and  work  by  Grossberg  (1989) .  All  of 
the  factors  were  evaluated  for  Importance  (on  a  7-point  scale) ,  and 
every  sector  in  each  specialization  area  was  rated  on  all  24 
factors  (on  7 -point  scales) . 

2.5  Results. 

The  results  consisted  of  several  different  sets  of  data  rating 
sector  complexity.  Some  data  had  been  collected  using  direct 
methods,  and  some  using  indirect  measures.  ,  Each  type  of 
information  was  considered  separately,  and  then  combined,  in  an 
attempt  to  identify  important  complexity  factors  for  the 
Jacksonville  ARTCC  airspace. 

2t,gt.Ji _ Sector  Sorts. 

The  five  controllers  within  each  specialization  area  had  sorted  the 
sectors  within  their  area  by  level  of  complexity.  Kendall 
correlations  were  calculated  for  each  area,  as  shown  in  table  3. 
This  statistic  indicated  the  degree  of  agreement  between  the  five 
FPLs  or  Developmentals  in  each  specialization  area.  The  final  line 
in  table  3  shows  the  amount  of  agreement  between  the  East  FPLs  and 
Developmentals.  In  every  case,  there  was  a  significant  level  of 
agreement,  suggesting  that  the  participants  sorted  the  sectors  by 
complexity  in  similar  ways. 
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Table  3.  Kendall  Coefficient  of  Concordance  for  Each 
Specialization  Area,  Developmentals, 
and  all  East  Controllers 


AREA 

w 

chi2 

df 

SIGNIFICANCE 

NORTH 

23.71 

7 

.0013 

SOUTH 

19.29 

6 

.0037 

EAST 

.77 

27.07 

7 

.0003 

.84 

25.04 

6 

.0003 

■KO 

23.66 

6 

.0006 

1  DEVELOPMENTAL 

HQ 

22.4 

7 

.0022 

■oa 

46.67 

7 

.0000 

The  following  sectors  were  judged  most  complex  in  each 
Specialization  Area: 


AREA 


SECTOR 


North 

South 

East 

West 

Central 

East  (Developmentals) 

2.5.2  Complexity  Factor  Lists. 


Brxinswick 

St .  Augustine 

St.  Johns  (tied) 

Aiken 

Crestview 

Seminole 

Aiken 


All  PPLs  were  asked  to  list  the  complexity  factors  they  thought 
were  most  important  in  evaluating  Jacksonville  ARTCC  sectors.  This 
resulted  in  over  200  factors.  The  lists  were  combined  and 
redundancies  were  removed.  Table  4  shows  the  resulting  shortened 
list  of  factors,  with  the  accompanying  frequency  and  percentage  of 
occurrence  of  each  factor.  Developmentals  were  excluded,  because 
it  was  found  that  much  of  their  data  diverged  from  that  of  the 
FPLs.  This  was  probably  due  to  their  relative  lack  of  experience. 

2.5.3 _ Complexity  Factor  Importance. 


Participants  had  also  rated  the  importance  of  the  24  Initial 
Factors,  in  order  to  determine  which  were  most  salient. 
Developmentals  were  again  excluded  from  the  combined  data  set,  due 
to  their  relatively  limited  amount  of  ATC  knowledge.  (It  was  noted 
that  eliminating  their  scores  from  the  data  substantially  reduced 
the  variability  in  the  ratings  for  each  factor.)  A  mean  importance 
rating  was  calculated  for  each  factor.  Table  5  shows  the  factors 
arranged  in  order  of  average  importance. 
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Table  4.  Factors  Listed,  Nvunber  of  Times  Mentioned,  and  Percent 
of  Possible  Responses  by  FPLs  During  Main  Study 


FACTOR 

FREQUENCY 

PERCENT 

CCM4PLEX  FLIGHT  PATHS 

30 

14.0 

RESTRICTED  AREAS,  WARNINGS,  MOA 

22 

5.5 

SIZE  AIRSPACE 

21 

5.6 

TRAFFIC  VOLUME 

18 

■■■EEl 

WEATHER 

16 

4.7 

ARRIVAL/DEPARTURE  SEQUENCING/ SPACING 

15 

4.6 

COORDINATION 

15 

4.9 

MILITARY  TRAFFIC 

14 

4.8 

AIRCRAFT  MIX 

13 

4.7 

SECTOR  BOUNDARIES 

13 

4.9 

FLOW  RESTRICTIONS 

10 

4.0 

TRANSITIONING 

9 

3.7 

CLIMBING/DESCENDING  FLIGHTS 

6 

2.6 

DEPARTING/ARRIVING  AIRCRAFT 

3 

1.3 

EQUIPMENT 

3 

1.3 

FREQUENCY  CONGESTION 

2 

0.9 

IRREGULAR  ALTITUDE  STRATIFICATION 

2 

0.9 

VFR/IFR 

2 

0.9 

MULTIPLE  FUNCTIONS 

1 

0.5 

Tabltt  5.  Average  Importance  Ratings  for  24  Complexity  Factors 


COMPLXXZTY  FACTOR 

AVBRA61 

RATING 

CLIMB/DESCEND  FLTS . 

6.3 

RESTRICTED  AREAS 

6.2 

WEATHER 

6.0 

NO.  FACILITIES 

5.9 

COORDINATION 

5.8 

RADAR/ RAD 10  COVERAGE 

5.6 

NO.  DEP/ARR  AIRCRAFT 

5.5 

AIRCRAFT  MIX 

5.4 

MULTIPLE  FUNCTIONS 

5.4 

RADIO  CONGESTION 

5.1 

INTERSECTING  FLIGHT  PATHS 

5.0 

COMPLEX  CONTROL  ADJ. 

5.0 

ADEQUACY  OF  EQUIP. 

5.0 

TRAFFIC  DENSITY 

4.8 

FREQ.  MIL.  REQS. 

4.7 

STARS/SIDS 

4.5 

COMPLEX  FLT.  PATHS 

4.5 

AIRLINE  HUBBING 

4.5 

SECTOR  BOUNDARIES 

4.5 

SIZE  AIRSPACE 

4.3 

FREQUENT  HOLDS /STRETCHING 

3.9 

FREQ.  PROC.  ADJ. 

3.9 

IRREG.  ALT.  STRAT. 

3.2 

SECTOR  FLIGHT  TIME 

3.1 
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The  Kendall  correlation  for  FPL  factor  importance  ratings  was  x^(23, 
N  *  24)  s  146.83,  p  «  .0000.  This  suggests  that,  although 
participants  were  from  different  specialization  areas,  there  was  a 
high  degree  of  agreement  among  them  about  the  importance  of  the 
initial  set  of  complexity  factors.  Controllers  from  separate  areas 
appeared  to  share  similar  viewpoints  regarding  complexity. 

2.1 5.1  i _ Sftgtor..Bfttjlnqg. 

FPLs  rated  sectors  within  each  specialization  area  on  each  of  24 
Initial  Factors,  compiled  dviring  the  preliminary  study.  These 
ratings  were  used  when  interpreting  MDS  and  Pathfinder  data. 
Product>moment  correlations  were  calculated  within  each 
specialization  area  to  check  on  the  agreement  between  participant 
ratings.  All  correlations  were  significant  at  p  <  .01. 

Data  for  all  5  specialization  areas  were  combined,  and  average 
ratings  on  each  of  the  24  factors  were  computed  across  participants 
for  all  38  sectors  in  the  Jacksonville  ARTCC.  (This  resulted  in 
each  sector  having  ratings  on  each  of  the  24  factors.) 
Developmentals '  data  were  excluded  from  this  analysis  under  the 
assumption  that  their  level  of  knowledge  was  not  comparable  with 
the  FPLs '  knowledge . 

A  weighted  rating  score  for  each  sector  was  calculated  by 
multiplying  the  rating  of  each  sector  on  each  of  the  24  factors  by 
the  average  importance  assigned  to  that  factor,  and  summing  across 
all  24  factors  for  each  sector.  (For  each  sector:  factor  1  rating 
X  Importance  rating  of  factor  1  plus  factor  2  rating  X  importance 
rating  of  factor  2,  etc.)  The  resulting  list  is  shown  in  table  6. 

This  list  also  includes  the  original  set  of  FAA  complexity  ratings 
from  the  1991  Annual  Review  of  Jacksonville  ARTCC  Sectors.  A 
comparison  of  the  order  of  the  sectors  in  each  list  (using  a 
Kendall  correlation)  showed  a  significant  relationship  (T  *  .38,  p 
»  .  000) ,  indicating  a  partial  matching  of  sector  complexity  ratings 
based  on  the  two  different  approaches. 

_ Complexity  Similarity  Ratings. 

There  was  l  set  of  proximity  measures  for  each  of  the  5  FPLs  from 
each  of  the  5  specialization  areas,  for  a  total  of  25  matrices,  and 
1  set  from  each  of  the  5  Developmentals.  There  were  also  six  sets 
of  proximity  measures  for  most  of  the  sectors  from  the  APSs  and  TMU 
staff. 

MDS  was  applied  to  the  controller  and  APS/TMU  similarity  judgments 
to  explore  the  structure  of  the  complexity  judgment  data.  The  FPL 
and  Developmental  data  will  be  discussed  separately  from  the  APS 
and  TMU  results,  given  that  much  smaller  sets  of  proximity  measures 
(i.e.,  similarity  judgments)  were  collected  from  the  former  groups. 
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TABLE  6.  WEIGHTED  RATINGS  OF  JACKSONVILLE  ARTCC  SECTORS. 
EACH  RATING  IS  THE  SUM  OF  THE  PRODUCTS  OF  AVERAGE  RATING 
OF  EACH  SECTOR  AND  THE  AVERAGE  RATED  IMPORTANCE  OF  THE 

FACTOR. 

ALSO  SHOWN  ARE  THE  FAA  COMPLEXITY  RATINGS  FROM  THE  1991 
ANNUAL  REVIEW  OF  JACKSONVILLE  ARTCC  SECTORS. 


2.S.6  MDS. 

2. 5. 6.1  PPL  and  Developaental  D&ta. 

INDSCAL  was  usad  to  analyze  the  complexity  similarity  data. 

INDSCAL  produces  a  common  stimulus  space  sheured  by  the 

participants,  and  indicates  Individual  differences  in  their  use  of 
the  identified  dimensions.  In  the  case  of  the  FPLs  and 

Developmentals ,  solutions  were  restricted  to  two  dimensions, 
because  of  the  limited  number  of  stimuli.*  Interpretation  of  the 
KDS  stimulus  spaces  for  each  of  the  specialization  areas  was 
accomplished  by  using  sector  descriptions  and  ratings  of  each 

sector  on  the  previously  discussed  initial  set  of  24  complexity 
factors.  Copies  of  the  stimulus  spaces  were  also  sent  to 

Jacksonville  ARTCC  personnel  for  their  review  and  interpretation. 

Dimensions  were  named  by  determining  whether  the  arrangement  of 
sectors  along  each  MDS  dimension  corresponded  to  the  increase  or 
decrease  in  some  variable  or  factor  related  to  complexity.  In  some 
cases,  interpretation  was  difficult;  nevertheless,  it  was  possible 
to  label  most  of  the  dimensions.  Details  of  these  analyses  are 
found  in  appendix  B. 

2. 5. 6. 2  SuMSirv  of  Dlaensiona. 

Listed  below  are  the  dimensions  identified  by  means  of  MDS  for  each 
of  the  specialization  areas.  Each  dimension  is  assumed  to 
represent  a  potential  complexity  factor.  (See  table  7  for  a  list 
of  the  factors,  with  redundancies  removed.) 


Table  7.  MDS  Factors  from  FPLs  in  Five  Specialization  Areas 
and  Frequency  of  Occurrence  of  Each  Factor 


rXCTOR 

FREQ. 

TYPE  OF  TRAFFIC  (OVERFLIGHTS  VS.  CLIMB/DESCEND) 

3 

VOLUME 

2 

VFR  VS.  IFR 

2 

NUMBER  OF  TRAFFIC  DIRECTIONS  (CROSSING  VS.  ONE-WAY) 

1 

NUMBER  OF  PROCEDURES 

1 

MULTIPLE  FUNCTIONS 

1 

MILITARY  TRAFFIC 

1 

FREQUENCY  CONGESTION 

1 

EFFECTS  OF  TRAFFIC  LOAD  (MANAGEABLE  VS.  HECTIC) 

1 

AMOUNT  OF  COOUJINATION 

1 

ALTITUDE  (LOW.  MORE  COMPLEXITY) 

1 

6  The  ■iallarity  judgment  data  were  aeaumed  to  be  at  the  ordinal  level  of 
measurement;  therefore,  a  non-metric  MDS  model  waa  employed.  Although  it  could 
be  argued  that  the  data  were  measured  at  the  interval  level,  the  more 
conservative  approach  of  assuming  the  ordinal  level  was  chosen.  The  use  of 
metric  versus  non-metric  MDS  did  not,  in  fact,  make  a  great  difference  in  the 
resulting  stimulus  ^ace  configurations. 
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NORTH 


1.  Frequency  congestion 

2.  Effects  of  traffic  load  (manageable  vs.  hectic) 

SOUTH 

1.  Visual  Flight  Rules  (VFR)  vs.  Information  Flight  Rules 
(IFR) ,  non-professional  vs.  professional  pilots 

2.  Volume,  type  of  traffic  (overflights  vs. 
arrivals/departures) 


EAST 

1.  Number  of  traffic  directions  (crossing  vs.  one-way) 

WEST 

1.  Volxime,  military  traffic 

2.  Type  of  traffic  (overflights  vs.  arrivals),  VFR  vs.  IFR, 
number  of  procedures,  altitude  (high  vs.  low) 

CENTRAL 

1.  Amount  of  coordination,  multiple  functions 

2.  Type  of  traffic  (overflights  vs.  climb/descend) 

DEVELOPMENTALS 

1.  West  vs.  East,  crossing  traffic  vs.  one-way  traffic 

2.  Large  vs.  small  airspace,  low  vs.  high  problems  with 
sector  boundaries 

2. 5. 6. 3  Airspace  Procedures  and  TMU  Personnel. 

The  data  sets  for  Airspace  Procedures  and  TMU  participants  were 
combined  for  analysis  using  INDSCAL.  Since  different  participants 
made  comparisons  between  different  sets  of  sectors,  depending  upon 
their  feuniliarity  with  the  airspace,  a  common  set  had  to  be  derived 
for  the  Airspace  Procedures  and  TMU  INDSCAL  analyses.  The  TMU 
participants  had  19  sectors  in  common,  but  the  APSs  had  only  9 
sectors  in  common.  These  common  sector  sets  were  used  in  the 
INDSCAL  analysis.  The  details  of  these  MDS  analyses  are  found  in 
appendix  B.  The  resulting  sets  of  complexity  factors  are  shown  in 
tables  B1  and  B2. 
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The  WSST  is  a  measure  of  cognitive  style  (Galin  and  Ornstein, 
1974) .  It  assesses  whether  an  individual  tends  to  solve  problems 
using  verbal  or  analytical  thinking,  as  opposed  to  spatial  or 
global  thinking.  The  total  possible  score  is  60.  A  high  score 
indicates  a  preference  for  verbal /analytical  processing,  while  a 
low  score  suggests  a  preference  for  spatial/global  processing.  A 
score  of  30  means  that  there  is  no  strong  tendency.  The  test  was 
used  in  an  attempt  to  explain  individual  differences  in  similarity 
judgments. 

A  number  of  analyses  showed  that  there  were  no  differences  in 
cognitive  style  between  the  various  controller  groups,  and  that 
WSST  scores  did  not  appear  to  explain  any  differences  in  MDS 
results  between  controllers.  (See  appendix  C  for  details.) 


The  preliminary  and  main  studies  produced  three  kinds  of  data  about 
sector  complexity,  from  three  sources.  The  first  kind  of  data  %ras 
the  rankings  of  sectors  for  complexity,  which  were  available  from 
the  sector  sorts.  These  rankings  helped  determine  the  most  complex 
sectors  in  the  Center,  and  they  can  be  compared  to  the  overall 
ratings  of  sectors  on  the  24  Initial  Factors.  The  second  kind  of 
data  were  the  factor  names  listed  by  the  preliminary  and  main  study 
participants.  The  frequency  with  which  factors  were  mentioned  on 
these  lists  can  be  compared  between  the  preliminary  and  main 
studies,  and  contrasted  to  the  ratings  of  factor  Importance 
collected  during  the  main  study. 

The  third  kind  of  data  also  identified  factors,  but  came  from  a 
different  source:  the  complexity  similarity  judgments.  Although 
sorts  and  lists  were  relatively  direct  methods  of  gathering 
information  on  sector  complexity,  similarity  judgments  were  the 
outcome  of  a  more  indirect  approach.  Use  of  this  method  assiuned 
that  valuable  Insights  about  complexity  factors  might  be  obtained 
by  asking  participants  to  make  comparisons  that  drew  upon  their 
knowledge  of  the  Jacksonville  sectors.  What  they  may  not  have  been 
able  to  report  verbally,  they  may  have  been  able  to  demonstrate  in 
their  judgments.  The  judgments  themselves  were  submitted  to  MDS 
analysis,  and  factor  names  were  developed  on  the  basis  of  the 
dimensions  that  emerged  from  the  MDS  results.  Accordingly,  the  MDS 
results  will  be  discussed,  in  addition  to  the  data  obtained  from 
the  sorts  and  lists. 


2.6.1  Sector  Complexity  and  Difficulty. 

One  hypothesis  of  this  study  was  that  sector  complexity  and  sector 
difficulty  would  be  highly  related,  with  difficulty  being  the  air 
traffic  controller's  subjective  response  to  complexity.  This  was 
confirmed  by  the  high  correlation  of  APS  ratings  of  sector 
complexity  and  difficulty  in  the  preliminary  study.  Therefore,  the 
sector  complexity  factors  derived  from  this  study  should  assist  in 
determining  which  sectors  in  the  Jacksonville  ARTCC  are  mcst 
complex  and  most  difficult  to  work. 

2.6.2  Sector  Sorts . 

The  preliminary  study  was  a  convincing  demonstration,  on  a  limited 
basis,  of  substantial  agreement  between  controllers  about  the 
complexity  of  sectors,  as  demonstrated  in  a  sorting  exercise. 
Correlations  of  sector  sorts  between  individual  controllers  from 
the  West  Specialization  Area  were  significant,  and  the  degree  of 
overall  agreement  between  them  was  also  significant. 

This  was  also  the  case  in  the  main  study  where  FPLs  and 
Developmentals  sorted  sectors  within  their  specialization  areas. 
Degree  of  agreement  among  rankings  was  significant  within  each 
area.  The  sector  judged  most  complex  by  the  four  West  Area  FPLs  in 
the  preliminary  study  (Crestview)  also  attained  the  highest  average 
rank  in  the  main  study,  where  similar  sorts  were  completed  by  five 
different  West  Area  controllers. 

Upon  comparing  the  ordering  of  sectors  from  FPL  sorts  in  each  area 
to  the  sectors  obtaining  the  highest  weighted  sum  of  factor  ratings 
in  the  main  study  (table  6) ,  it  was  found  that  two  of  the  sectors 
at  the  top  of  the  sector  sorts,  Crestview  (West)  and  Aiken  (East) , 
were  also  on  the  list  of  the  five  highest-rated  sectors.  nie 
highest-rated  sectors  from  the  other  three  specialization  areas 
were  found  in  lower  ordinal  (sorted)  positions.  In  the  South  Area, 
St.  Johns  and  St.  Augustine  (tied  for  first  by  sorting)  were 
ranked  seventh  and  eleventh  (respectively)  on  the  rating  list; 
Brunswick  (North)  was  ninth;  and  Seminole  (Central)  was  twelfth. 

A  more  comprehensive  check,  on  the  correspondence  of  the  aurea  sorts 
and  the  ratings  of  sectors  on  factors,  was  conducted  by  noting,  for 
the  sectors  in  each  specialization  area,  the  order  of  their 
appearance  in  table  6  (weighted  sector  ratings) .  The  purpose  of 
this  check  was  to  evaluate  the  comparability  of  the  two  methods: 
sorting  and  rating.  Kendall  correlations  were  then  calculated, 
comparing  the  sort  order  with  the  rating  order  for  each  area.  The 
results  are  shown  in  table  8. 
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Tabl«  8.  Correlation  of  Sector  Rankings  with  Ratings 
of  Sectors  on  Preliminary  Complexity  Factors 


SPSC.  ARIA 

KENDALL ’ S  TAU 

SIGN. 

NORTH 

.71 

.005 

SOOTH 

.95 

.002 

EAST 

.62 

.007 

WEST 

.98 

.001 

CENTRAL 

.39 

.390 

All  but  one  of  the  correlations  in  table  8  were  significant.  The 
correlation  coefficients  for  the  South  and  West  Specialization 
Areas,  in  particular,  indicated  a  high  level  of  agreement.  In  the 
case  of  the  West  Area,  for  example,  the  complexity  rankings  from 
sorting  would  have  been  identical  to  those  for  the  complexity-based 
ratings,  except  that  two  sectors  in  the  sorted  list  were  tied. 
These  results  suggest  that  the  two  methods  of  determining  sector 
complexity  usually  yield  comparable  results. 

2.6.3  Complexity  Factors . 

The  first  attempt  (in  the  preliminary  study)  to  gather  complexity 
factors,  by  simply  asking  APSs  and  FPLs  to  list  them,  produced  61 
factors.  Because  some  of  these  were  repeated  more  or  less  verbatim 
by  several  participants  (see  table  1) ,  it  was  possible  to  reduce 
the  list  to  19  factors.  The  first  7  factors  accounted  for  64 
percent  of  the  61  factors  mentioned.  The  first  factor  (Restricted 
Areas,  Warning  Areas,  MOAs,  and  Notification  Time)  was  mentioned  by 
8  of  the  10  respondents  (for  13  percent  of  the  total) . 

A  second  list  of  215  complexity  factors  was  gathered  during  the 
main  study.  Removal  of  redundancies  resulted  in  a  list  of  19 
factors  as  found  in  table  4.  A  comparison  of  tables  1  and  4  shows 
that  the  following  factors  are  near  the  top  on  both  lists:  Complex 
Flight  Paths;  Restricted  Areas,  Warning  Areas,  and  MOAs;  Weather; 
Size  of  Airspace;  and  x'raffic  Volume.  However,  a  comparison  of 
tables  4  and  5  (ratings  of  factor  importance)  does  not  show  much 
agreement,  except  about  Restricted  Areas,  Warning  Areas,  and  MOAs; 
Weather;  and  Coordination. 

It  might  be  assumed  that  the  frequency  of  a  factor's  being 
mentioned  was  related  to  its  importance  to  the  group  of  FPLs. 
Given  the  above  results,  this  assumption  seems  uncertain,  and  it 
was  investigated  further  in  section  2.8. 
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2.6.4  Complexity  Similarity  Judgements. 

In  the  preliminary  study,  the  four  FPLs  were  asked  to  compare  pairs 
of  West  Area  sectors  for  similarity  in  complexity.  Overall  tests 
of  the  agreement  of  the  four  participants  indicated  a  significant 
level  of  accord. 

The  high  correlations  of  similarity  between  individual 
participants'  judgments  for  photographs  and  maps,  indicated  that 
the  photographs  and  maps  of  sectors  were,  for  all  Practical 
purposes,  interchangeable  as  the  basis  for  collecting  complexity 
similarity  judgments.  Although  controllers  from  different  areas  of 
the  Center  tended  to  prefer  the  representation  they  worked  with 
most  often,  photos  and  maps  both  seemed  to  provide  adequate  stimuli 
for  the  recall,  not  only  of  the  physical  attributes  of  the 
airspjice.  but  also  of  the  typical  traffic  patterns,  weather 
activity,  and  so  forth.  Participants  discussed  these  various 
aspects  of  sectors  when  presented  with  photographs  or  maps  of  the 
ctirspace  (without  traffic) . 

The  INDSCAL  MDS  spaces  for  the  West  Area  photographs  and  maps  were 
very  rimilar.  In  spite  of  the  differences  between  individual  sets 
of  similarity  ratings,  the  MDS  INDSCAL  software  was  able  to  derive 
a  shared  stimulus  space  that  accounted  for  91  percent  of  the 
variability  in  the  set  of  judgments.  Thus,  the  degree  of 
concordance  between  the  four  FPLs,  which  emerged  when  the  judgments 
themselves  were  compared,  was  reflected  in  a  well-fitting  shared 
stimulus  space. 

In  the  main  study,  as  in  the  preliminary  study,  there  was  generally 
a  high  degree  of  agreement  about  which  sectors  within  each 
specialization  area  were  similar  in  complexity.  The  only  exception 
was  the  East  Area,  where  the  level  of  controller  agreement  was 
marginal . 

Combining  the  sector  complexity  similarity  judgments,  within  each 
specialization  area  using  INDSCAL,  was  successful  in  that  the 
common  stimulus  space  for  each  group  efficiently  accounted  for 
controller  data  in  each  case. 

Upon  considering  individuals  within  each  area,  it  was  found  that 
the  amount  of  variance  in  judgments,  accounted  for  by  INDSCAL,  was 
generally  above  0.60.  However,  some  participants  in  each  area  were 
not  well  represented  by  the  common  MDS  space,  indicating  that  there 
was  not  always  agreement  within  each  group  regarding  the  complexity 
factors  used  to  make  the  similarity  judgments.  For  those 
participants  who  were  adequately  represented,  there  were 
differences  in  each  controller's  emphasis  on  the  identified  common 
dimensions. 
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The  intention  of  collecting  data  on  thinking  style  (in  this  case, 
the  WSST)  was  to  assist  in  explaining  individual  differences  in 
participants'  cognitive  structures  and  decision-making  patterns. 
At  this  stage  of  the  study,  the  primary  information  on  variations 
in  cognitive  structxires  emerged  from  the  INDSCAL  weight  spaces. 
These  data  Illustrated  how  heavily  each  participant  used  the  common 
dimensions  identified  for  the  group  of  five  controllers  within  each 
specialization  area.  However,  it  appears  that  cognitive  style  (as 
measured  by  the  WSST)  did  not  account  for  individual  differences  in 
knowledge  structures  in  this  case. 

It  was  possible  to  compare  the  MDS  results,  for  the  preliminary  and 
main  studies,  only  for  the  West  Area.  The  description  of  the  two 
dimensions  in  each  solution  was  almost  identical.  The  close 
correspondence  of  the  MDS  dimensions  from  two  different  groups  of 
West  Area  FPLs  suggests  that  the  procedure  is  a  reliable  means  for 
identifying  complexity  factors. 

The  list  of  dimensions,  collected  from  the  interpretations  of  the 
FPL  MDS  spaces  in  the  main  study,  contained  only  three  factors  that 
were  repeated  more  than  once.  Type  of  Traffic  (overflights  vs. 
climbing/descending} ,  Traffic  Volume,  and  VFR  vs.  IFR  traffic 
appeared  three,  two,  and  two  times,  respectively.  The  remaining 
eight  factors  appeared  only  once  in  the  list  (table  7) .  Due  to  the 
differences  in  the  characteristics  of  sectors  in  each 
specialization  area,  it  is  understandable  that  there  might  have 
been  variations  in  complexity  factors. 

It  is  interesting  to  note  that  there  was  some  correspondence  in  the 
East  FPL  and  Developmental  MDS  results.  (The  Developments Is  were 
being  trained  in  the  East  Area.)  This  suggests  that  it  might  be 
possible  to  use  MDS  as  a  method  for  assessing  progress  in  training. 
Training  success  might  be  indicated  when  Developmentals '  cognitive 
structures  (as  measured  by  MDS)  begin  to  resemble  those  of  their 
instructors.  This  method  of  assessing  training  effectiveness  could 
be  used  to  complement  other  methods.  It  is  attractive  in  that  it 
does  not  depend  on  supervisor  ratings. 

The  INDSCAL  stimulus  spaces  for  the  Airspace  Procedures  and  TMU 
participants  accounted  for  a  large  proportion  of  the  information  in 
the  original  similarity  judgments,  and  each  participant  was  well 
represented.  Participants  differed  to  some  extent  in  their  use  of 
the  available  dimensions  in  making  their  judgments.  The 
statistical  approach  employed  for  interpretation  resulted  in  many 
factors  being  identified,  especially  in  the  three-dimensional  TMU 
stimulus  space.  Each  of  the  property  vectors’  was  effective  in 
locating  meaningful  complexity  factors,  and  did  an  excellent  job  of 
explaining  the  pattern  of  sectors  in  the  MDS  stimulus  space. 


7  See  appendix  B  for  an  explanation  of  property  vectors. 
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Upon  comparing  the  property  vectors  for  the  Airspace  Procedures  and 
TMU  INDSCAL  stimulus  spaces  (tables  B1  and  B2) ,  it  was  discovered 
that  all  but  one  of  the  vectors  identified  for  the  APS  INDSCAL 
solution  were  found  in  the  TMU  results.  The  missing  factor. 
Coordination,  is  closely  related  to  Number  of  Facilities,  and  was 
one  of  the  factors  present  in  the  TMU  vectors.  Thus,  there  was 
strong  agreement  between  the  APS  and  TMU  participants  regarding 
complexity  factors. 

Of  the  11  factors  that  emerged  from  the  INDSCAL  results  for  the  5 
specialization  areas  (table  7) ,  6  matched  those  found  in  the  total 
of  18  APS  or  TMU  property  vectors  (tables  B1  and  B2) ,  as  shown  : 


Table  9.  Relationship  Between  Specialization  Area  Factors 

and  APS/TMU  Factors. 


SPEC.  AREA  PACTOR 

APS/TMU  FACTOR 

Type  of  traffic 

Climbing/Descending  Flights 

VFR  vs.  IFR 

Aircraft  Mix 

No.  of  Traffic  Directions 

Intersecting  Flight  Paths 

No.  of  Procedures  and  Multiple 
Functions 

Multiple  Functions 

Military  Traffic 

Frequent  Military  Requirements 

Coordination 

Coordination 

The  most  noticeable  omissions  (factors  present  in  the  FPL  MDS 
results  but  absent  from  the  APS/TMU  lists)  were  Frequency 
Congestion,  Volume,  and  Altitude.  Niuober  of  Procedures  was  also 
missing,  but  is  probably  closely  related  to  Multiple  Functions. 
Considering  that  different  approaches  were  used  to  derive  the  sets 
of  MDS  dimensions,  a  respectable  degree  of  agreement  nevertheless 
emerged. 

2.6.5  Comparison  of  Direct  and  Indirect  Methods. 

A  comparison  of  the  MDS  results  (tables  7,  Bl,  and  B2) ,  with  the 
compilation  of  factor  lists  found  in  table  4,  shows  that  eight  of 
the  first  nine  factors  of  table  4  are  also  found  in  the  MDS 
results.  These  factors  accounted  for  48  percent  of  those 
mentioned.  Table  10  lists  the  factors  from  table  4,  and  their 
correspondence  with  the  MDS  results. 
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In  sc»M  cases,  factors  that  occurred  frequently,  or  were 
particularly  salient  property  vectors  in  the  FPL,  APS,  and  TMU  MDS 
results,  appeared  low  in  frequency  of  occurrence  on  the  susuBary  of 
factor  lists  (table  4) .  Examples  are  VFR  vs.  IFR  Traffic,  Number 
of  Departing/Arriving  Aircraft,  Multiple  Functions,  and  Climbing/ 
Descending  Flights. 

A  factor  that  appeared  in  three  cases  in  the  FPL  MDS  dimensions  was 
not  included  in  the  factor  lists  from  table  4;  this  was.  Type  of 
Traffic  (Overflights  vs.  Arrivals) .  However,  t^is  could  be  related 
to  Number  of  Climbing  vs.  Descending  Flights.  Factors  from  the 
Airspace  Procedures /TMU  MDS  results,  which  did  not  appear  in  table 
4,  included  Airline  Hubbing,  Number  of  Facilities,  and  presence  of 
Standard  Terminal  Arrival  Routes  (STARS)  and  standard  Instrument 
Departures  (SIDS) .  On  the  other  hand,  several  factors  mentioned 
fairly  often  in  table  4  did  not  emerge  in  the  MDS  results.  These 
included  Sector  Boundaries,  Flow  Restrictions,  and  Transitioning 
Traffic.  Although  there  was  some  agreement  between  the  direct  and 
indirect  approaches,  some  factors  emerged  from  each  source  that 
were  not  common  to  both. 


Table  10.  Comparison  of  Factors  from  Lists  and  Factors 
from  FPL,  APS,  and  TMU  MDS  Results 


FACTORS  FROM  LISTS 

MDS 

CORRESPONDSNCB 

COMPLEX  FLIGHT  PATHS 

FPL,  TMU 

RESTRICTED  AREAS,  WARNINGS,  MOAS 

SIZE  AIRSPACE 

TMU 

TRAFFIC  VOLUME 

FPL 

WEATHER 

TMU 

ARRIVAL/DEPARTURE  SEQUENCING/SPACING 

APS,  TMU 

COORDINATION 

FPL,  APS 

MILITARY  TRAFFIC 

FPL,  TMU 

AIRCRAFT  MIX 

APS,  TMU 

SECTOR  BOUNDARIES 

FLOW  RESTRICTIONS 

TRANSITIONING 

CLIMBING/DESCENDING  FLIGHTS 

APS,  TMU 

DEPARTING /ARRIVING  AIRCRAFT 

APS,  TMU 

EQUIPMENT 

TMU 

FREQUENCY  CONGESTION 

FPL 

IRREGULAR  ALTITUDE  STRATIFICATION 

VFR  VS.  IFR 

FPL,  APS,  TMU 

MULTIPLE  FUNCTIONS 

FPL,  APS,  TMU 
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Based  on  these  sources,  it  was  possible  to  generate  a  list  of 
complexity  factors  for  further  analysis.  First,  the  following 
factors,  shared  by  at  least  two  of  the  three  MDS  lists  (tables  7, 
B1  and  B2)  were  considered  important  and  were  included: 

a.  Type  of  Traffic  (overflights  vs.  climbing /descending) 

b.  Aircraft  Mix  (VFR  vs.  IFR) 

c.  Intersecting  Flight  Paths 

d.  Multiple  Functions 

e.  Military  Traffic 

f .  Coordination 

In  addition,  one  item  in  table  7  emerged  more  than  once,  but  was 
not  in  the  common  list:  traffic  volume. 

The  following  items  in  tables  B1  and  B2  appeared  meaningful,  but 
were  not  in  the  above  lists: 

a.  Number  of  Facilities. 

b.  Number  of  Departing/Arriving  Aircraft. 

c.  Airline  Rubbing. 

d.  Weather. 

The  direct  approach  resulted  in  several  factors  being  mentioned 
frequently  in  table  4,  but  not  included  in  the  MDS-derived  lists: 

a.  Complex  Flight  Paths. 

b.  Restricted  Areas,  Warning  Areas,  and  MOAs. 

c.  size  of  Airspace. 

d.  Arrival  Departure  Sequencing/ Spacing. 

e.  Equipment  or  Radio/Radar  Coverage  (also  appeared  in 
table  B2) . 


f .  Frequency  Congestion  (also  appeared  in  table  7) . 

To  these  were  added  two  other  factors  assumed  to  be  highly 
correlated  with  those  listed.  This  was  done  to  determine  whether 
they  were  independent  factors  or,  as  expected,  highly  related  with 
those  already  under  consideration.  At  the  recommendation  of 
Jacksonville  ARTCC  staff.  Manageability  of  the  Sector  under 
Increasing  Traffic  Load  (which  emerged  from  the  FPL  MDS  analysis) 
was  included  in  the  list.  It  was  assumed  that  this  factor  would  be 
nearly  identical  to  Traffic  Volume. 
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Center  staff  suggested  that  Multiple  Functions  and  Nusber  of 
Proc^ures  (both  of  which  emerged  in  the  FPL  MDS  results)  were 
probably  closely  related.  To  verify  this,  Nuxtber  of  Procedures  was 
added  to  the  list.  This  resulted  in  a  set  of  19  Candidate  Factors 
(as  sho%m  in  table  ll) ,  which  were  used  in  the  next  pairt  of  the 
research. 

2.7  INTERIM  PHASE  I  CONCLUSIONS. 

Ratings  of  sectors  on  complexity  and  difficulty  indicated  a  high 
degree  of  relationship  between  the  two  concepts.  A  complex  sector 
is  also  a  difficult  one.  The  development  of  improved  metrics  for 
assessing  complexity  will,  therefore,  be  helpful  in  identifying 
sectors  that  are  difficult  to  manage,  and  in  specifying  factors 
that  contribute  to  high  controller  workload. 

The  two  kinds  of  stimuli  (maps  and  photographs)  chosen  to  represent 
sectors  in  the  preliminary  study,  yielded  comparable  judgments  amd 
MDS  solutions.  This  finding  is  useful  for  future  work,  in  that, 
sector  maps  (without  traffic)  have  been  found  to  be  an  acceptable 
stimulus  for  making  comparisons,  and  gaining  other  information 
about  sectors.  Screen  photographs  could  also  be  used,  but  are 
initially  more  difficult  to  acquire. 

Controllers  from  the  five  specialization  areas  in  the  Jacksonville 
ARTCC  generally  agreed  about  overall  sector  complexity  as 
determined  by  sector  sorts.  In  addition,  the  order  of  complexity 
of  sectors,  within  each  area,  corresponded  well  with  the  ranking  of 
sectors  for  complexity,  based  on  a  set  of  ratings  gathered  from  the 
same  participants.  Sorting,  therefore,  appears  to  be  a  useful 
method  for  determining  complexity. 

Compiling  suggested  complexity  factors,  from  lists  by  study 
participants,  generated  two  lists,  one  from  the  preliminary  study, 
and  another  from  a  larger  number  of  controllers  in  the  main  study. 
There  was  some  correlation  between  the  lists,  and  the  main  study 
list  was  used  in  later  comparisons  with  the  MDS  results.  When 
factors  from  the  preliminary  study  list  were  ordered  by  ratings  of 
importance,  however,  there  was  not  much  agreement  with  the 
frequency  of  occurrence  of  factors  suggested  by  the  same 
participants.  This  finding  Indicates  that,  although  a  factor  might 
have  been  mentioned  frequently,  its  rate  of  occurrence  may  not  be 
an  accurate  indicator  of  its  importance. 
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Table  11.  List  of  Candidate  Sector  Complexity  Factors 


1. 


THE  RELATIVE  FREQUENCY  OF  OVERFLYING  VERSUS  CLIMBING/ 
DESCENDING  TRAFFIC.. _ 


2. 


THE  DEGREE  OF  AIRCRAFT  MIX  (VFR, 
TURBOPROPS.  JETS,  ETC.). _ _ 


IFR,  PROPS, 


13. 

4. 


THE  OCCURRENCE  OF  INTERSECTING  AIRCRAFT  FLIGHT  PATHS. 
THE  EXTENT  TO  WHICH  THE  CONTROLLER  MUST  PERFORM 
MULTIPLE  FUNCTIONS  (APPROACH  CONTROL,  TERMINAL  FEEDER, 
EN  ROUTE,  IN  TRAIL  SPACING.  ETC. ) _ 


THE  AVERAGE  AMOUNT  OF  MILITARY  TRAFFIC, 


6. 


THE  NEED  FOR  COORDINATION  (WITH  OTHER  SECTORS,  CENTERS, 
APPROACH  CONTROLS.  MILITARY  FACILITIES.  ETC.). _ 


7. 


THE  AVERAGE  VOLUME  OF  TRAFFIC. 


HOW  MANAGEABLE  A  SECTOR  IS  AS  TRAFFIC  VOLUME  INCREASES. 
THE  NUMBER  OF  DIFFERENT  PROCEDURES  A  CONTROLLER  MUST  BE 
FAMILIAR  WITH  AND  EMPLOY  IN  ORDER  TO  OPERATE  THE 
SECTOR^ _ 


10. 


THE  NUMBER  OF  FACILITIES  ACTUALLY  DEALT  WITH  UNDER 
NORMAL  CONDITIONS  (OTHER  SECTORS,  CENTERS,  APPROACH 
CONTROLS.  MILITARY  FACILITIES.  ETC.)  . _ 


11. 


THE  AVERAGE  NUMBER  OF  DEPARTING  AND  ARRIVING  AIRCRAFT , 
THE  EXTENT  TO  WHICH  THE  CONTROLLER'S  WORK  IS  AFFECTED 
BY  AIRLINE  RUBBING. _ _ _ 


12. 


13. 


THE  DEGREE  TO  WHICH  WEATHER  PROBLEMS  AFFECT  AIR  TRAFFIC 
CONTROL  OPERATIONS. _ _ _ _ 


14. 


THE  RELATIVE  FREQUENCY  OF  COMPLEX  AIRCRAFT  ROUTINGS  AS 


OPPOSED  TO  DIRECT  ROUTINGS. 

15. 

THE  EXTENT  TO  WHICH  THE  CONTROLLER'S  WORK  IS  AFFECTED 
BY  RESTRICTED  AREAS.  WARMING  AREAS,  AND  MOAs. 

nm 

THE  SIZE  OF  SECTOR  AIRSPACE. 

wsm 

THE  NEED  FOR  ARRIVAL /DEPARTURE  SEQUENCING  AND  SPACING. 

iOi 

THE  ADEQUACY  OF  RADIO  AND  RADAR  COVERAGE. 

19. 

THE  AMOUNT  OF  RADIO  FREQUENCY  CONGESTION  DURING  PEAK 
TRAFFIC  PERIODS. 

It  was  possible  to  deternine  the  most  complex  sector  in  each 
specialization  area  from  the  sector  sorts.  An  overall  ranking  of 
sector  complexity  for  all  38  Jacksonville  sectors  was  accomplished 
by  adding  up  ratings  on  24  Initial  Factors,  multiplied  by  the 
importance  rating  of  each  factor.  There  was  a  relatively  low  but 
significant  correlation  between  the  order  of  the  list  based  on 
factor  ratings,  and  the  list  from  the  PAA  annual  review.  However, 
this  correlation  was  not  strong  enough  to  indicate  a  meaningful 
relationship  between  the  two  complexity  measures.  Confirmation  of 
this  will  require  further  study. 

In  both  the  preliminary  and  main  studies,  similarity  judgments  of 
sector  complexity  indicated  some  individual  differences,  but  there 
was  sufficient  agreement  that  common  MDS  stimulus  spaces  could  be 
derived  for  each  specialization  area,  the  Developments Is,  and  the 
Airspace  Procedures /THU  staff.  Individual  differences  emerged, 
however,  in  the  weight  spaces  for  each  solution,  where  participants 
varied  in  the  amount  of  emphasis  they  placed  on  a  given  dimension. 
A  test  of  thinking  style,  the  WSST,  did  not  account  for  the 
differences  in  perception  of  sector  complexity. 

Different  approaches  (subjective  and  statistical)  were  used  to 
interpret  the  MDS  spaces  in  the  main  study.  When  comparing  the 
resulting  factors,  there  was  considerable  agreement  between  the  APS 
and  TMU  results,  both  of  which  used  the  statistical  approach. 
There  was  also  some  similarity  between  the  FPL  factors  and  the 
APS/TKU  data  (subjective  versus  statistical  approach) ,  although 
some  mismatches  were  evident. 

Emerging  from  this  work  is  a  list  of  potential  complexity  factors, 
some  of  which  were  common  to  both  direct  and  indirect  approaches, 
and  some  of  which  were  unique.  There  is,  as  yet,  no  information  to 
indicate  which  factors  are  most  important  in  accounting  for  sector 
complexity.  It  is  evident  that  some  of  the  factors  are  related. 
In  the  next  phase  of  the  study,  further  work  was  completed  to 
determine  which  Candidate  Factors  were  correlated,  and  how 
adequately  they  account  for  sector  complexity. 

2.8  FACTOR  SELECTION. 

The  goals  of  this  part  of  the  Phase  I  research  were  to  reduce  the 
19  Candidate  Factors  to  a  more  compact  set,  and  to  verify  that  the 
reduced  set  of  factors  could  adequately  account  for  overall  sector 
complexity. 
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Volunteer  participants  included  25  FPLs,  5  TMU  controllers,  and  a 
controller  from  Military  Operations.  FPL  participants  were  dra%m, 
based  on  availability,  from  those  working  on  the  control  room  floor 
on  the  days  of  the  researchers'  visits  to  the  Center.  Five  of  the 
PPL  participants  had  also  participated  in  the  factor  development 
phase  of  the  study.  The  other  participants  were  selected  on  the 
basis  of  their  availability;  three  of  these  individuals  had  also 
participated  in  factor  development. 


Data  collection  materials  for  the  FPLs  included  3  kinds  of  paper 
forms  designed  to  collect  ratings  of  the  sectors  in  each 
specialization  area  on  the  19  Candidate  Factors  listed  in  table  11. 
Ratings  were  needed  to  assess  the  importance  of  the  factors,  and  to 
rank  the  sectors  on  complexity.  Information  was  also  collected  on 
participant  age  and  years  of  experience.  TMU  and  supervisory 
controllers  used  a  different  form,  which  was  designed  to  record 
ratings  of  overall  complexity  on  all  38  sectors  in  the  Jacksonville 
ARTCC. 


The  FPL  forms  were  combined  into  a  package,  and  administered  to 
participants  who  were  avalled>le  from  the  control  floor.  Five  FPLs 
from  each  specialization  area  first  sorted  maps  of  the  sectors 
comprising  their  area  into  order  of  overall  complexity.  They  then 
rated  the  sectors  in  their  areas  on  all  of  the  19  Cemdidate 
Factors.  Finally,  they  evaluated  the  importance  of  each  factor  on 
a  seven-point  scale.  THU  participants  completed  their  ratings  in 
their  offices.  They  rated  all  the  sectors  in  the  center  on  overall 
complexity.  In  each  case,  the  researchers  provided  instructions 
and  answered  questions,  as  needed. 


In  order  to  check  on  the  agreement  between  the  5  FPLs  in  each 
specialization  area,  product-moment  correlations  of  the  sector 
ccmplexity  ratings  using  the  19  Candidate  Factors  were  calculated. 
In  every  area,  except  one,  correlations  between  participants  were 
significant.  In  the  Central  Specialization  Area,  however,  one 
participant's  ratings  were  significantly  correlated  with  two  of  the 
others,  but  not  with  the  other  two  controllers.  After  analysis  of 
the  data,  it  was  decided  to  retain  this  participant's  ratings  for 
further  analysis.  (Removing  this  participant  from  the  analysis  did 
not,  in  fact,  substantially  alter  the  results  reported  below.) 


The  six  TMU  and  supervisory  controllers  (who  were  faniliar  with  the 
Center's  airspace  in  general),  rated  all  38  sectors  in  overall 
complexity.  The  product-moment  correlations  between  their  ratings 
were  significant,  except  for  one  case.  One  rater's  results  did  not 
correlate  with  any  of  the  others.  Further  investigation  revealed 
that  five  of  the  six  controllers  who  rated  the  overall  complexity 
of  the  sectors  were,  or  had  recently  been,  employed  in  the  TMU, 
where  they  performed  flow-control  functions.  One  controller, 
however,  was  working  in  Military  Operations  and  had  not  controlled 
air  traffic  for  several  years.  It  was  this  participant's  ratings 
which  did  not  correlate  with  the  other  controllers'  evaluations  of 
overall  sector  complexity.  Therefore,  this  rater's  results  were 
excluded  from  the  set,  and  the  judgments  of  the  five  remaining  mu 
personnel  were  averaged  to  create  a  measure  of  overall  sector 
complexity.  This  measure  was  called  the  "Complexity  Criterion." 

2.9.2  Importance  of  Factors. 

The  next  step  was  to  identify  the  most  important  subset  of  the  19 
Candidate  Factors  for  determining  airspace  complexity.  First,  an 
overall  sector  complexity  score  was  calculated  by  averaging  the 
ratings  each  sector  received  from  5  FPLs  on  each  of  the  19 
Candidate  Factors.  All  19  scores  were  then  added  together  for  each 
sector,  resulting  in  a  "Sector  Complexity  Index."  By  combining  the 
results  for  sectors  from  all  specialization  areas,  a  set  of 
Complexity  Indices  for  all  38  sectors  in  the  Center  was  developed. 

When  the  set  of  complexity  indices  derived  from  all  19  factors  was 
correlated  with  the  Complexity  Criterion  provided  by  the  5  TMU 
controllers,  a  significant  product-moment  correlation  was  obtained 
(r  «  .63,  p  <  .01).  This  finding  indicated  that  a  simple,  non- 
weighted  sum  of  all  of  the  Candidate  Factors  was  moderately 
successful  in  accounting  for  the  information  in  the  Complexity 
Criterion.  It  was  reasonable  to  assume,  however,  that  a  subset  of 
these  factors  would  do  an  equal  or  better  job  of  accounting  for 
overall  complexity. 

As  a  basis  for  identifying  such  a  subset  of  factors,  separate 
correlations  were  calculated  between  the  set  of  sector  scores  on 
each  of  the  19  Candidate  Factors,  and  the  set  of  Complexity 
Criterion  scores.  (That  is,  between  scores  for  all  38  sectors  on 
factor  1  and  Complexity  Criterion  scores  for  the  same  sectors,  then 
for  factor  2  and  criterion  scores,  etc.)  The  results  provided  an 
ordering  of  the  complexity  factors  by  the  size  of  their  product- 
moment  correlations  with  the  criterion  (table  12)  .  It  was  evident 
that  the  first  factor  alone.  Routings  (the  proportion  of  complex, 
as  opposed  to  direct,  aircraft  routings)  ,  had  a  higher  correlation, 
with  overall  complexity,  than  complexity  indices  based  on  the  sum 
of  all  19  factors.  Those  11  Candidate  Factors,  with  significant 
simple  correlations  with  overall  sector  complexity,  were  called 
Final  Factors. 
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Table  12.  Product-Moment  Correlations  of  Sector  Complexity 
Scores  with  the  Complexity  Criterion  on  38  JacJcsonville 
Sectors.  (The  "NO.”  column  refers  to  the  full 
description  of  the  variables  in  Table  7.  "NS”  denotes 
the  lack  of  a  significant  correlation.) 


MO. 

VARZABLS 

PRARSOM  r 

SZGMZrZCANCS 

14 

ROUTINGS 

.73 

p  <  .01 

17 

SPACING 

.68 

p  <  .  01 

19 

CONGEST 

.62 

p  <  .01 

4 

FUNCTION 

.56 

p  <  .01 

3 

INTERSECT 

.53 

p  <  .01 

7 

VOLUME 

.52 

p  <  .01 

9 

PROCEDURE 

.42 

p  <  .01 

12 

HUBBING 

.40 

p  <  .  05 

6 

COORDINATE 

.37 

p  <  .05 

13 

WEATHER 

.37 

p  <  .05 

8 

MANAGVOL 

.34 

p  <  .05 

1 

OVERFLY 

.32 

NS 

11 

DEPARR 

.29 

NS 

18 

COVERAGE 

.24 

NS 

C 

MILITARY 

.15 

MS 

15 

RESTRICT 

.10 

NS 

10 

FACILITY 

.04 

NS 

2 

MIXTURES 

-.03 

NS 

16 

SECSIZE 

-.14 

NS 
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Multiple  regression*  was  then  used  to  determine  which  Final  Factors 
were  most  salient  in  accounting  for  overall  sector  complexity  (see 
table  13) .  Although  table  12  shows  that  seven  of  these  factors  had 
significant  individual  correlations  with  the  complexity  Criterion, 
they  were  also  correlated  with  each  other  (or  were  redundant  to 
some  degree) .  Table  14  shows  a  correlation  matrix  of  all  the  Final 
Factors.  The  regression  analysis  took  these  correlations  into 
account  and  estimated  only  the  unique  contribution  (or  lack 
thereof)  of  each  variable  in  accounting  for  overall  sector 
complexity.  The  resulting  multiple  correlation  (R  =  .85)  was  quite 
high,  indicating  that  the  identified  variables  were  successful  in 
predicting  overall  sector  complexity. 

FPLs  from  each  specialization  area  had  also  rated  the  importance  of 
the  19  complexity  factors.  There  was  a  significant  level  of 
overall  agreement  between  the  controllers  on  the  importance  of  the 
Candidate  Factors  as  evaluated  by  a  Kendall  correlation  x^(25,  N  - 
5)  «  98.19,  p  «  .0000.  It  was  found  that  the  importance  ratings 
were  not  effective  in  identifying  any  factors  related  to  overall 
complexity. 

Given  the  above  results,  it  appears  that  overall  sector  complexity, 
as  defined  by  the  ratings  of  a  group  of  TMU  controllers,  can  be 
predicted  by  as  few  as  three  of  the  Candidate  Factors.  These  are 
Routings  (the  proportion  of  complex,  as  opposed  to  direct,  aircraft 
routings) ;  Spacing  (need  for  arrival /departure  sequencing  and 
spacing) ;  and  Congestion  (radio  frequency  congestion) .  This 
finding  suggests  that  estimates  of  sector  complexity  could  be  made 
on  a  very  straightforward  basis,  using  only  a  limited  number  of 
factors. 


2.9.3  Factor  Intercorrelations . 

It  should  not  be  assumed,  however,  that  these  factors  are  the  only 
ones  involved  in  complexity.  As  table  12  demonstrates,  several 
other  Final  Factors  were  also  strongly  related  to  overall 
complexity.  For  example,  the  regression  fojrmula  using  the  next 
three  most  highly  correlated  factors  (Number  of  Multiple  Functions, 
Traffic  Volume,  and  Number  of  Intersecting  Airways)  with  overall 
complexity  was  also  effective  in  predicting  complexity.  Routings, 
Spacing,  and  Congestion  were  highly  correlated  with  other  factors 
and  can  be  considered  representative  of  them  in  the  regression 
equation  reported  in  table  13.  These  intercorrelations  indicate 
that  many  of  the  complexity  factors  tended  to  occur  together  in 
complex  sectors  in  the  Jacksonville  ARTCC. 


8  Multiple  regression  is  a  statistical  technique  that  draws  a  relationship 
between  a  set  of  measures  or  variables  and  some  other  variable  of  interest.  A 
formula  is  created  that  shows  the  proportion  of  each  of  the  "predictor”  variables 
that  oiust  be  added  together  to  best  predict  or  account  for  the  "outcome" 
variable. 
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Table  13.  Multiple  Regression  Results  using  the  Complexity 
Criterion  as  the  Dependent  Measure  and  Complexity 
Factors  as  Independent  Variables 


Multiple  R 

.85 

R  Square 

.72 

Adjusted  R 

Square 

.69 

Standard  Error 

.55 

Analysis  of 

Variance 

DF 

Sum  of 

Squares 

Mean 

Square 

Regression 

3 

26.45 

8.82 

Residual 

34 

10.38 

.31 

r  -  28.88 

Slgnif  F  -  .0000 

Variable 

B 

SC  B 

Beta  T 

Sig.  T 

ROUTINGS 

.44 

.13 

.39 

3.35 

.0020 

SPACING 

.23 

.07 

.37 

3.47 

.0015 

CONGEST 

.32 

.12 

.29 

2.77 

.0087 

(Constant) 

-.17 

.56 

-.31 

.76 

Table  14.  Product-Moment  Correlations  Between  Final  Factors. 
(*  -  Significant  at  p  <.05,  **  -  Significant  at  p  <  .01) 


CONGEST 


CROINATE  FUNCTION  HUBBZNG 


ZNTBRSCT  IMANAGVOI. 


Tabl«  15.  Most  Isportant  Coaplexity  Factors,  the  Basis  for  their 
Measurement,  and  their  Appearance  on  Lists  of  Candidate 
Factors  using  Direct  and  Indirect  Approaches  in  Phase  I. 

(An  N/A  is  shown  where  it  was  not  possible  for  the 
variable  to  emerge  in  the  MDS  analysis  given  the 
interpretation  method  involved.) 


SECTOR 

MSASURXMBHT 

FPL 

LIST 

FPL 

MDS 

APS 

MDS 

TMU 

MDS 

ROUTINGS 

SPECIALIST  OPINION 

X 

N/A 

N/A 

SPACING 

BASED  ON  SECTOR  DATA 

X 

CONGEST 

BASED  ON  ANALYSIS  OP  VOICE 
TAPES 

m 

Hi 

FUNCTION 

BASED  ON  SECTOR  DATA 

X 

X 

X 

X 

INTERSECT 

BASED  ON  SECTOR  DATA 

X 

X 

VOLUME 

BASf.D  ON  SECTOR  DATA 

X 

X 

PROCEDURE 

SPECIALIST  OPINION 

X 

RUBBING 

BASED  ON  SECTOR  DATA 

X 

SPECIALIST  OPINION 

X 

X 

X 

WEATHER 

BASED  ON  SECTOR  AND  WEATHER 
DATA 

■i 

m 

MANAGVOL 

SPECIALIST  OPINION 

X 

N/A 

N/A 

During  the  factor  identification  process,  two  factors  included  as 
candidates  for  fuxrther  study  were  assumed  to  be  highly  correlated 
with  other  factors  in  the  list.  These  were  Manageable  Volume 
(thought  to  be  correlated  with  Volume)  and  Number  of  Procedures 
(thought  to  be  correlated  with  Multiple  Functions)  .  When  product- 
moment  correlations  were  calculated  between  all  of  the  Candidate 
Factors,  it  was  found  that,  contrary  to  predictions,  the 
correlation  between  Manageable  Volume  and  Volume  was  not 
significant.  Apparently  the  niunber  of  aircraft  typically  found  in 
a  sector  is  not  necessarily  related  to  how  difficult  the  sector  is 
to  control  as  traffic  increases.  Observations  indicated  that  the 
interaction  of  traffic  volume  and  other  sector  characteristics, 
such  as  sector  size,  determined  difficulty.  As  predicted,  there 
was  a  high  correlation  between  Number  of  Procedures  and  Multiple 
Functions.  It  appears  that  these  two  measures  are  strongly 
related. 

Other  significantly  related  measures  from  the  correlation  matrix 
included  Type  of  Traffic  (overflights  vs.  climbing/descending)  and 
Number  of  Departing/ Arriving  Aircraft;  Type  of  Traffic  and  Multiple 
Functions;  Manageable  Volume  and  Multiple  Functions;  Number  of 
Procedures  and  Manageable  Volume;  and  Aircraft  Mix  and  Military 
Traffic.  Sector  Size  and  Type  of  Traffic  were  strongly  negatively 
correlated.  (Values  of  "r"  and  obtained  probability  levels  can  be 
inspected  in  table  14.) 
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The  correlations  between  the  19  Candidate  Factors  listed  in  table 
14  were  used  as  input  to  Pathfinder.  As  previously  discussed,  a 
Pathfinder  network  consists  of  a  set  of  concepts  and  links  that 
directly  connect  pairs  of  highly-related  concepts.  It  was  assumed 
that  using  correlational  data  with  Pathfinder  would  assist  in  the 
understanding  of  the  relationships  between  the  complexity  factors, 
and  would  throw  light  on  controller  knowledge  structures. 

Figure  5  shows  the  Pathfinder  network  derived  from  the  inter-factor 
correlations  found  in  table  14.  Since  correlations,  as  opposed  to 
proximities  (or  similarities)  were  used  to  generate  the  network, 
the  links  in  the  graph  represent  degree  of  correlation  or  co¬ 
occurrence  of  the  factors  in  Jacksonville  ARTCC  sectors. 

Inspection  of  figure  5  indicates  three  main  clusters  of  factors. 
Starting  at  the  top,  the  number  of  MOAs  and  Restricted  Areas  is 
related  to  the  number  of  Military  Operations.  Military  Operations, 
Facilities,  and  the  nximber  of  Departing  and  Arriving  Aircraft  are 
related  to  the  Degree  of  Aircraft  Mix.  The  number  of 
Climbing/ Descending  (as  opposed  to  overflying)  Flights  are  linked 
to  the  presence  of  Departing  and  Arriving  Aircraft.  These  factors 
are  also  connected  with  the  need  for  the  controller  to  perform 
Multiple  Functions. 

Toward  the  center  of  figure  5  is  a  tight  grouping  of  six  factors, 
including:  !!ultiple  Functions,  Complex  Routings,  Number  of 
Procedures,  Need  for  Interfacility  Coordination,  Manageable  Volume, 
and  Number  of  Intersecting  Airways.  This  clustering  indicates  that 
these  sector  characteristics  tend  to  occur  together.  Just  below 
this  set  are  two  factors  involving  the  Need  for  Arrival /Departure 
Sequencing  and  Spacing  and  Radio/Radar  Coverage.  Spacing  is 
related  to  Airline  Hubbing.  This  last  factor  appears  to  be  part  of 
another  grouping  at  the  bottom  of  the  graph  along  with  Traffic 
Volume,  Frequency  Congestion,  Sector  Size,  and  Weather. 

There  is  one  central  node  in  each  of  the  major  clusters.  (The 
factors  enclosed  in  boxes  on  a  Pathfinder  graph  are  called  nodes. 
Nodes  are  like  atoms  one  sees  in  a  model  of  a  complex  molecule.  A 
central  node  is  one  that  is  linked  to  several  other  nodes.)  The 
central  nodes  obtained  in  this  analysis  are  Aircraft  Mix,  Multiple 
Functions,  and  Traffic  Volume.  These  nodes  may  represent  a  primary 
characteristic  of  three  general  types  of  complex  sectors,  as 
defined  by  the  three  major  clusters  identified  in  this  analysis. 

When  the  Final  Factors  are  mapped  onto  the  Pathfinder  network,  they 
occupy  all  but  2  of  the  13  factors  beneath  (but  not  including) 
Overfly  on  figure  5.  (Final  Factors  are  shaded  in  figure  5.)  All 
of  the  factors  in  the  central  group  are  included  in  the  Final 
Factors  list,  indicating  that  this  constellation  of  complexity 
characteristics  may  be  critical  in  defining  sector  complexity  as 
experienced  by  en  route  controllers. 
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Figure  5.  Pathfinder  Network  Diagram  based  on  the 
Correlations  of  19  Complexity  Factors 
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The  choice  of  factors  employed  to  estimate  sector  complexity  should 
not  be  based  only  upon  their  simple  or  multiple  correlation  with 
the  Complexity  Criterion,  but  also  on  their  ease  and  reliability  of 
measurement.  Discussions  with  the  Jacksonville  ARTCC  staff 
resulted  in  the  list  shown  in  table  15.  This  list  specifies  those 
factors,  taken  from  the  Final  Factors,  that  Center  personnel  could 
feasibly  measure,  even  if  they  were  not  intimately  familiar  with 
all  sectors  in  the  airspace. 

Although  all  of  the  factors  are  potentially  measurable,  measurement 
of  some  factors  would  clearly  require  more  effort  than  others. 
Those  marked  as  "specialist  opinion"  need  the  involvement  of  FPLs 
from  the  specialization  area  of  concern.  Evaluation  of  a  sector 
using  the  other  Final  Factors  could  be  handled  by  generalist 
personnel  (such  as  airspace  planners) ,  with  sufficient  data,  as 
required . 

When  the  factors  requiring  specialist  opinion  were  eliminated  from 
the  list,  and  those  remaining  were  entered  into  a  multiple 
regression  procedure,  it  was  still  possible  to  predict  overall 
complexity  with  reasonable  accuracy  using  only  two  factors. 
Arrival /Departure  Sequencing  and  Spacing  and  Frequency  Congestion 
(R  =  .78,  p  <  .0000)  .  This  type  of  approach  might  make  it  possible 
for  generalists  to  assess  sector  complexity. 


one  of  the  purposes  of  this  research  was  to  evaluate  direct  versus 
indirect  approaches  for  exploring  controller  knowledge  about  sector 
complexity.  The  direct  approach  involved  asking  controllers  direct 
questions  about  sector  complexity;  the  indirect  approach  was  based 
on  statistical  analysis  of  controllers'  judgments  of  sectors 
similarity  in  complexity.  Armed  with  the  ordering  of  the 
complexity  factors  found  in  table  12,  it  was  possible  to  revisit 
the  factor  development  stage  of  this  study  to  determine  whether  one 
or  the  other  approach  showed  any  advantage.  The  results  of  this 
investigation  are  also  shown  in  table  15. 

One  of  the  Final  Factors,  Arrival/Departure  Sequencing  and  Spacing, 
was  mentioned  in  the  FPL  lists,  but  was  not  found  in  the  MDS 
results.  Three  Final  Factors  that  emerged  from  the  MDS  results 
were  not  found  in  the  combined  FPL  lists:  Intersecting  Flight 
Paths,  Number  of  Procedures,  and  Airline  Hubbing.  Although  these 
three  complexity  factors  did  not  appear  on  the  lists  given  by  FPL 
controllers,  they  were  suggested  during  the  preliminary  study  by 
other  controllers.  These  results  indicate  that,  in  many  cases, 
direct  methods  may  be  adequate  for  knowledge  exploration. 


47 


Ci 


The  formula  for  the  regression  equation  in  table  13  was  used  to 
calculate  complexity  indices  for  all  of  the  Jacksonville  sectors. 
(Ratings  for  each  sector  on  the  factors  in  the  equation  were 
entered  in  the  formula  in  order  to  generate  an  estimate  of 
complexity  for  each  sector.)  These  Indices  were  then  correlated 
with  the  Center  complexity  ratings,  based  on  the  FAA  formula 
contained  in  FAA  Order  7210.46.  The  regression-based  complexity 
indices  had  low  correlations  with  the  FAA  scores.  There  were 
insufficient  operational  error  data  from  the  Center's  records  to 
determine  whether  or  not  there  was  any  relationship  between  the 
regression-based  indices  and  operational  errors. 

2.9.7  Refinement  of  Factors. 

Finally,  the  19  Candidate  Factors  under  consideration  were  reviewed 
by  a  Jacksonville  ARTCC  controller  who  had  been  extensively 
involved  in  the  project.  This  controller  and  the  researchers 
reviewed  the  factors  and  data  describing  their  relative  importance 
and  intercorrelations.  This  team  found  that  it  was  possible  to 
modify  the  list  by  combining  redundant  factors,  and  improving  the 
description  of  each  item.  The  product  was  a  list  of  15  factors 
with  definitions  and  descriptions  of  their  contributions  to 
complexity,  as  follows: 

1.  The  amount  of  climbing  or  descending  traffic.  Climbing  and 
descending  aircraft  are  those  that  are  transitioning  altitudes, 
including  departure  and  arrival  traffic,  and  aircraft  requesting 
altitude  changes  due  to  turbulence,  pilot  preference,  etc.;  or 
aircraft  that  require  different  altitudes  to  alleviate  conflictions 
due  to  crossing  traffic  or  other  problems.  Climbing  and  descending 
traffic  medces  maintaining  separation  more  complicated,  and 
increases  the  niunber  of  actions  the  controller  must  take  and 
monitor . 

2.  The  degree  of  aircraft  mix  fVFR,  IFR.  props,  turboorops.  iets. 
etc . ) .  The  performance  characteristics  of  jets  versus  propeller 
aircraft,  as  well  as  pilot  capabilities,  can  affect  the  work  of  the 
controller.  Traffic  mix  can  create  problems  due  to  jets  overtaking 
propeller  aircraft,  limited  climb  capabilities  of  propeller 
aircraft,  differences  in  skill  between  VFR  and  IFR  pilots,  or  VFR 
pilots  encovmtering  IFR  conditions.  Such  conditions  create 
additional  workload  for  the  controller. 

3.  The  number  of  intersecting  flight  paths.  This  factor 
represents  the  number  of  inherent  converging  flight  paths  due  to 
airways,  arrival  routes,  or  frequent  requests  for  direct  routings. 
Converging  flight  paths  increase  the  chance  of  conflictions. 
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A  qg*  approach  control,  terminal  feeder,  en  r^oute.  in-trail 
spacing,  etc.).  An  ATC  function  is  a  related  set  of  tasks  or 
services  performed  by  the  controller.  A  sector  that  requires 
numerous  functions  requires  the  controller  to  be  familiar  with,  and 
perform  the  tasks  associated  with  each  function. 

5.  The  number  Of  required  procedures  that  must  be  performed 
(example:  all  Raleigh  arrivals  must  croga  TB?NI  _at  FL2ia) .  A 
procedure  is  a  group  of  tasks,  or  a  specific  task,  required  by 
regulation  or  direction.  A  procedure  mandates  controller  actions 
and  must  be  performed  regardless  of  other  required  tasks. 

6.  The  number  of  military  flights.  Military  flights  include  all 
types  of  military  aircraft.  Military  flights  produce  complexity 
due  to  differences  in  performance  characteristics  and  nature  of 
mission  requirements.  Military  pilots  also  have  a  tendency  to  meJce 
special  requests. 

7.  Amount  of  coordination  or  interfacing  with  other  entities 
(such  as  adjacent  sectors,  approach  controls,  center,  military 
units,  etc. \ .  Coordination  is  commvtnication  with  other  controllers 
or  facilities  to  discuss  critical  information  regarding  air 
traffic.  Coordination  requires  fzunlliarity  with  contacting 
methods,  applicable  directives  outlining  procedures  between  the 
facilities,  and  methods  for  performing  coordination  activities.  In 
addition,  coordination  must  be  performed,  as  required  by  the 
demands  of  the  traffic  in  the  sector. 

8.  The  extent  to  which  the  controller  is  affected  bv  airline 
hubbinq  or  major  terminal /airport  traffic.  Airports  differ  in  the 
density  of  traffic  flow  in  and  out  of  a  geographical  area.  Hubbing 
refers  to  the  practice  of  airlines  using  a  central  airport  for 
their  regional  operations.  Hubbing  and  heavy  airport /terminal 
traffic  creates  a  focused  concentration  of  flights  in  one  area  with 
the  attendant  increase  in  risk  of  separation  conflicts  and  number 
of  controller  tasks. 


9-  The  extent  to  which _ weather-related  factors  affect 

operations.  Weather  includes  (but  is  not  limited  to) 
thunderstorms,  turbulence,  icing,  precipitation,  convective 
activity,  or  IFR  conditions.  Weather  affects  complexity  because 
pilots  are  unwilling  or  unable  to  fly  into  certain  weather 
conditions.  This  causes  requests  for  deviations  from  procedures 
and  flight  paths,  as  well  as  other  changes. 


10.  Number  of  complex  aircraft  routings.  A  simple  routing  is  in 
effect  when  an  aircraft  flies  along  its  planned  flight  path. 
Complex  routings  occur  when  aircraft  frequently  require  manual 
controller  input  such  as  vectors,  altitude  changes,  large  course 
changes,  intercepts,  etc.,  or  require  close  monitoring.  Complex 
routings  deviate  from  the  normal,  and  require  more  attention  due  to 
Increased  chances  for  conflict ions. 

11.  The^  extent  to  which  the  controller's  work  is  affected  bv 
regtricted_areaa^  _  warning  areas,  and  MOAs  and  their  associated 
activities .  Once  a  restricted  area  is  activated,  no  flights  are 
allowed  in  or  out  of  a  that  airspace.  VFR  aircraft  are  allowed  to 
fly  in  warning  areas  (over  the  ocean)  or  into  HOAs,  although  this 
is  not  advisable.  These  areas  reduce  the  eunount  of  airspace 
available  to  controllers,  and  create  obstructions  to  flight  routes. 
This  increases  the  likelihood  of  conf lictions  because  aircraft  must 
be  rerouted  around  them.  Activities  around  these  airspaces  are 
specialized  flights  requiring  special  handling  and  monitoring. 

12.  The  size  of  sector  airspace.  Size  of  airspace  refers  to  the 
volume  of  airspace  contained  within  the  lateral  and  horizontal 
boundaries  of  the  sector.  Size  affects  complexity  because  a  small 
sector  has  less  airspace  for  the  controller  to  utilize  in  conflict 
resolution.  Size  limits  the  controller's  ability  to  handle  traffic 
volume,  and  deal  with  any  special  conditions,  such  as  weather. 

13.  The_^^cruirement  for  longitudinal  sequencing  and  spacing. 
Sequencing  refers  to  prioritizing  the  order  of  aircraft  for  arrival 
over  a  fix  or  destination.  Spacing  refers  to  the  distance  created 
or  required  between  each  aircraft  in  the  sequence.  Sequencing  and 
spacing  increase  the  requirement  for  planning  and  can  increase 
conflict  potential,  since  they  require  the  streaming  of  aircraft 
from  several  sources  into  one  streeun. 

14 .  AdeguacY  and  reliability  of  radio  and  radar  coverage.  Some 
low  sectors  lack  sufficient  radar  coverage  to  detect  aircraft  below 
4000  feet,  and  some  lack  the  radio  coverage  needed  to  enable 
reliable  voice  communication  in  all  portions  of  the  sector.  When 
radar  coverage  is  inadequate,  the  controller  must  revert  to  work¬ 
intensive,  non-radar  procedures.  Insufficient  radio  coverage 
requires  the  use  of  alternate  communication  technicpies,  such  as 
pilot-to-pilot  relays.  Loss  of  radio  and  radar  coverage  can  result 
in  a  need  for  search-and-rescue  operations. 


15.  Amount  of  radio  freguencv  congestion.  Radio  frequency 
congestion  can  result  from  a  high  number  of  aircraft  in  a  sector, 
or  can  be  affected  by  numerous  pilot  requests  due  to  turbulence  or 
other  factors,  such  as  emergencies.  Complexity  results  because 
congestion  limits  the  controller's  ability  to  utilize  the  frequency 
for  issuing  instructions  to  aircraft.  Increased  probability  of 
separation  loss  and  additional  effort  to  monitor  or  communicate 
with  aircraft  can  also  result. 


In  all  phases  of  this  study,  there  was  considerable  evidence  that 
controllers  within  each  specialization  area  agreed  on  their 
evaluation  of  the  sectors  faAiliar  to  them.  In  the  preliminary 
study,  four  West  Area  controllers  agreed  on  their  ordering  of 
sectors  by  complexity,  and  were  generally  close  on  their  ratings  of 
similarity  of  complexity.  During  factor  development,  there  was  a 
significant  correlation  between  ratings  FPLs  made  on  sectors  in 
their  areas  on  an  initial  set  of  24  complexity  factors. 
Controllers  were  also  generally  in  agreement  regarding  whi^ 
sectors  were  similar  in  complexity,  and  each  group's  judgments 
could  be  adequately  accounted  for  by  INOSCAL-generated,  shared 
stimulus  spaces. 

When  rating  the  sectors  on  a  reduced  set  of  19  Candidate  Factors 
during  factor  selection,  all  peurticipants  within  each 
specialization  area  (with  the  partial  exception  of  1  Central  Area 
controller)  were  in  agreement.  Of  the  six  controllers  who  rated 
overall  sector  complexity,  five  made  similar  evaluations  of  the 
sectors.  The  one  who  did  not  lacked  recent  exposure  to  ATC 
operations. 

In  attempting  to  identify  the  most  effective  set  of  factors  for 
evaluating  sector  complexity,  it  was  found  that  11  Final  Factors 
had  significant  simple  correlations  with  the  Complexity  Criterion. 
Multiple  regression  revealed  that  a  combination  of  three  of  these 
factors  (the  Proportion  of  Complex  vs.  Direct  Aircraft  Routings, 
the  Need  for  Arrival /Departure  Sequencing  and  Spacing,  and  Radio 
Frequency  Congestion)  were  strongly  related  to  overall  complexity 
and  could  presumably  be  used  to  estimate  the  complexity  of  sectors 
in  the  Jacksonville  ARTCC.  However,  the  numerous  correlations 
between  the  11  Final  Factors  suggested  that  they  all  play  some  role 
in  sector  complexity,  and  that  any  theoretical  account  of  the 
sources  of  sector  complexity  should  include  them.  In  fact,  given 
the  methods  used  to  collect  the  19  Candidate  Factors,  all  should  be 
considered  as  potential  variables  for  future  research  in  this  area. 

On  a  practical  note,  some  of  the  Final  Factors  could  only  be 
measured  by  controllers  trained  in  the  airspace  of  concern. 
However,  the  analysis  identified  a  subset  of  the  these  factors, 
which  could  be  easily  evaluated  by  generalists  using  sector  maps, 
weather  information,  and  traffic  counts.  Two  out  of  this  set  were 
combined  in  an  equation  to  adequately  account  for  overall  sector 
complexity.  These  factors  were  Spacing  and  Frequency  Congestion.’ 


9  However,  it  would  not  be  wise  to  use  a  small  number  of  factors  to  assess  sector 
complexity.  More  reliability  would  be  accomplished  using  a  number  of  readily 
measurable  vauriables. 
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'nius,  there  was  evidence  that  a  workable  method  could  be  developed 
to  allow  Jacksonville  controllers  (such  as  those  in  the  Airspace 
Planning  and  Procedvures  Office)  to  evaluate  sector  complexity 
without  having  extensive  )cnowledge  about  all  of  the  sectors  in  the 
Center . 

The  analysis  revealed  that  indices  of  sector  complexity,  using  the 
pareuaeters  from  the  multiple  regression  equation,  were  closely 
related  to  the  Complexity  Criterion  provided  by  TMU  controllers. 
However,  neither  of  these  sets  of  ratings  correlated  with  FAA 
complexity,  as  determined,  using  the  rules  contained  in  FAA  Order 
7210.46.  Although  the  FAA  rating  system  may  not  adequately  account 
for  sector  complexity,  further  validation  work  is  needed  to  confizm 
that  the  method  for  evaluating  sector  complexity  generated  by  this 
study  is  more  effective. 

One  purpose  of  this  study  was  to  evaluate  direct  and  indirect 
techniques  for  studying  ATC  knowledge  structures.  As  noted  in  the 
introduction,  a  case  can  be  made  for  including  a  technique  like 
MDS,  if  it  is  assumed  that  requesting  data  in  the  form  of  judgments 
(which  rely  on  an  underlying  knowledge  base)  can  allow  access  to 
information  that  is  not  easily  verbalized.  Experience  with  both 
techniques  demonstrated  that  the  selected  indirect  approach  was  a 
useful  source  of  candidate  sector  complexity  factors,  but  was  time 
and  labor  Intensive  in  its  application.  The  final  results  showed 
that  MDS  did  not  identify  any  unique  complexity  factors;  all  of  the 
Final  Factors  were  directly  sugge3ted  by  controllers  at  some  point 
in  the  study. 

Thus,  the  indirect  approach  for  knowledge  elicitation  used  in  this 
research,  did  not  prove  to  have  any  distinct  advantages  in  terms  of 
information  yield  over  directly  asking  controllers  for  complexity 
factors.  While  the  exercise  of  combining  the  data  from  direct  and 
indirect  approaches  was  useful  for  selecting  prominent  factors  for 
further  analysis,  the  time  and  effort  required  to  collect  and 
interpret  the  MDS  data  must  also  be  considered.  MDS  should  be  used 
in  cases  where  multiple  sources  of  data  are  needed,  such  as  to 
reduce  a  large  list  of  potential  factors  by  seeking  overlapping 
information  from  unrelated  techniques.  In  other  cases,  it  may  be 
that  direct  knowledge-exploration  techniques  are  adequate  for  the 
purpose  of  identifying  items,  such  as  sector  complexity  factors. 

For  interpretation  of  MDS  results,  a  statistical,  as  opposed  to 
subjective,  method  of  interpretation  should  be  employed,  unless  the 
experimenters  have  direct  access  to  SMEs  in  the  target  domain.  By 
employing  previously  gathered  sets  of  ratings  in  a  regression 
analysis,  a  statistical  method  obviates  the  need  for  the  technical 
knowledge  required  for  direct  interpretation  of  MDS  spaces. 
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The  results  of  this  study  are  specific  to  the  Jacksonville  ARTCC, 
and  may  not  be  fully  generalizable  to  other  en  route  or  terminal 
ATC  facilities.  It  is  likely  that  some  of  the  Identified 
complexity  factors  are  common  to  many  areas,  although  some,  such  as 
certain  weather  problems,  may  be  unique  to  particular  en  route  and 
terminal  areas. 


Although  this  study  has  contributed  to  the  identification  of  the 
factors  that  are  important  in  creating  complex  airspace,  further 
validation  work  is  needed  to  confirm  these  results.  Ultimately,  a 
useful  tool  for  evaluating  sector  complexity  will  result.  Although 
controllers  showed  agreement  on  their  ratings  of  sector  complexity, 
the  data  revealed  that  they  did  not  necessarily  concur  on  the 
relative  importance  of  complexity  factors.  This  finding  suggests 
that  individual  controllers  may  respond  differently  to  a  given 
constellation  of  complexity  factors. 

Assuming  that  many  of  the  factors  contributing  to  sector  complexity 
have  been  delineated,  this  permits  further  work  on  the  effects  of 
complexity  on  controller  decision  making.  Accordingly,  this  report 
continues  with  a  description  of  the  Phase  II  research  on  controller 
decision  making. 


3.  PHASE  II  -  DECISION  MAKING  fPRELIMINARY  STUDYl. 
3.1  INTRODUCTION. 


The  preliminary  decision-making  study  was  designed  to  relate  the 
findings  on  sector  complexity  to  controllers'  behavior  «rhile 
managing  sector  traffic.  It  was  assumed  that  the  information 
collected  about  sector  complexity  factors  would  provide  a 
background  against  which  controller  decision-medcing  activities 
could  be  analyzed. 


3t2  MEROAgH. 

A  simple  decision-making  model  was  adopted,  based  on  one  documented 
in  an  en  route  ATC  training  manual  (FAA,  1989) .  This  model  is 
diagrammed  in  figure  6.  The  model  assumes  that  decision  making 
occurs  in  four  phases:  Scanning,  Projecting,  Planning,  and  Acting. 
The  Scanning  phase  involves  continuously  switching  attention 
between  different  kinds  of  visual  and  auditory  data.  Visual  data 
are  provided  by  the  radar  display  and  flight  progress  strips,  while 
auditory  data  are  available  from  radio  and  telephone  communication 
systems.  The  research  teeua  assimed  that  an  tinder  lying  mental 
representation  of  the  ATC  situation  directs  the  switching  of 
controller  attention  between  these  sources  of  data.  The  team 
further  assumed  that  the  purpose  of  scanning  is  to  update  this 
mental  representation  as  the  basis  for  detecting  potential 
problems . 
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6.  Decision-Making  Model 


According  to  the  adopted  models  upon  detecting  a  potential 
conflict,  the  controller  Projects  the  future  positions  and 
altitudes  of  the  aircraft  concerned,  and  then  assesses  the 
likelihood  of  the  conflict.  During  the  Planning  phase,  the 
controller  is  thought  to  generate  and  review  the  possible  solutions 
to  the  problem.  (An  important  alternative  is  recognition  of  the 
solution  based  on  past  experience,  which  does  not  involve 
generation  of  all  possible  solutions.) 

The  adopted  model's  Acting  phase  represents  the  decision  to  select 
and  implement  a  chosen  solution.  (if  the  solution  is  recognized, 
projecting  leads  directly  to  acting.)  The  Scan-Project-Plan-Act 
cycle  is  repeated  continuously  while  the  controller  is  working  air 
traffic.  While  this  model  is  primarily  a  heuristic  for  training 
purposes,  it  matches  controllers'  reports  of  their  decision-medcing 
process  sufficiently  to  guide  preliminary  research.  The  decision¬ 
making  study  employed  time-limited,  simulated  ATC  problems  to 
analyze  each  stage  of  this  initial  decision  model. 

Approaching  the  study  of  controller  decision  making  in  this  way 
allowed  information  to  be  collected  on  various  aspects  of  the 
process.  It  also  permitted  consideration  of  the  effects  of 
knowledge  structures  or  sector  complexity  factors  on  the  three 
major  phases:  Problem  Analysis  (including  Scanning  and  Projecting) , 
Planning,  and  Acting.  It  was  assumed  that  the  sector  complexity 
factors  liferent  in  each  ATC  problem  might  influence  controller 
behavior  in  observable  ways.  Measurement  of  such  effects  was 
restricted  to  a  qualitative  analysis.  At  this  stage,  no 
comparisons  were  made  between  sectors  or  problems  on  any  objective 
variables  that  might  be  correlates  of  decision  processes. 

3.3  METHOD. 

3.3.1  Participants . 

The  initial  set  of  participants  included  seven  FPLs  from  various 
specialization  areas  at  the  Jacksonville  ARTCC.  The  choice  of 
participants  was  restricted  to  supervisory  personnel  only.  Given 
this  participant  pool,  it  was  not  possible  to  limit  the  study  to 
controllers  from  one  specialization  area,  as  had  been  originally 
planned.  Instead,  participants  from  other  areas  were  Included; 
this  allowed  comparison  of  results  between  areas. 

There  were  three  North  Area  controllers,  two  from  South,  and  one 
each  from  the  Central  and  West  Areas.  Experience  as  an  FPL  ranged 
from  5  to  17  years.  However,  none  of  the  participants  was  working 
full  time  as  a  controller  at  the  time  of  the  study.  They  were 
functioning  in  supervisory  or  other  non-operational  positions.  All 
participants  but  one  maintained  minimum  required  ATC  currency  by 
controlling  traffic  a  minimum  of  16  hours  per  month. 
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The  preliminary  decision-making  study  was  conducted  on  a  4B6 
personal  computer  with  a  17-inch  color  display.  TRACX>N  (Wesson, 
1988)  ATC  simulation  software  was  used  to  create  six  air  traffic 
problems  ranging  from  low  to  high  complexity  (as  evaluated  using 
the  previously  discussed  complexity  factors)  in  three  of  the  North 
Specialization  Area  sectors.  Accurate  sector  maps  and  realistic 
ATC  problems  were  developed  with  the  assistance  of  a  North  Area 
controller.  One  of  these  problems  was  selected  and  refined  for  use 
during  the  experiment. 

Each  experimental  session  took  place  in  the  Computer-Based  Training 
Room  st  the  Center.  The  experimenter  collected  data  and  acted  as 
a  simulator  pilot  by  entering  participants'  ATC  instructions  into 
the  simulator  with  a  keyboard.  Sessions  were  recorded  on  video  and 
audio  tape. 


Participants  were  seated  in  front  of  the  computer  monitor  and  given 
an  introduction  to  the  experiment.  Then  a  simple  air  traffic 
problem  was  run  using  a  "generic  sector"  to  allow  practice  with  the 
simulation  program.  When  the  participant  indicated  readiness,  a 
problem  was  started  that  had  been  created  for  the  Florence  sector, 
in  the  North  Specialization  Area.  The  seven  aircraft  in  the 
problem  were  allowed  to  enter  the  sector  and  move  to  a  set  start 
point,  at  which  time  the  simulator  was  paused.  The  participant  was 
provided  with  flight  progress  strips  for  each  aircraft,  and  asked 
whether  sufficient  information  was  present  to  understand  the 
problem  and  formulate  a  solution. 

A  series  of  questions  was  reviewed  before  the  problem  was 
restarted.  To  address  Scan  and  Project  (Problem  Analysis)  issues, 
participants  were  first  asked  to  note  all  pairs  of  aircraft  that 
might  come  into  conflict,  assuming  the  problem  was  allowed  to  run 
ahead  with  no  controller  intervention.  Once  this  was  accomplished, 
the  controller  estimated  a  safety  risk  probability  (in  percent) 
with  each  conflict.  Participants  were  then  asked  to  provide 
comments  and  ratings  on  the  importance  of  data  from  the  radar 
screen,  as  opposed  to  flight  strip  data,  and  on  the  criticality  of 
each  category  of  aircraft  information  for  understanding  the 
problem.  (These  assessments  were  made  on  seven-point  scales.  It 
was  not  feasible  to  include  voice  communication  as  a  factor  because 
of  limitations  imposed  by  the  simulation  system.) 


Ratings  of  the  problem  on  each  of  13  of  the  previously-deve loped 
complexity  factors'”  were  requested,  and  each  participant  was  asked 
to  report  familiarity  with  the  sector,  and  problem,  and  to  estimate 
the  difficulty  of  the  problem  (on  seven-point  scales) .  The 
controller  responsible  for  generating  the  problem  was  also  asked  to 
evaluate  its  complexity  on  the  same  scale,  to  provide  a  baseline 
for  comparison. 

Each  controller  was  also  asked  whether  the  solution  to  the  problem 
had  emerged  fully  formed,  or  had  been  recognized  immediately, 
suggesting  recognition-primed  decision  making  (RPD)  (Klein,  1989) , 
or  whether  it  had  been  worked  out  from  first  principles.  RPD 
essentially  by-passes  the  generation  and  evaluation  of  alternative 
solutions,  going  directly  from  identification  of  the  problem  to 
acting.  RPD  is  characteristic  of  expert  problem  solving  in 
naturalistic  settings  (i.e.,  outside  the  laboratory). 

The  data  collected  on  the  Plan  portion  of  the  decision-making  model 
included  two  kinds  of  predictions  made  by  participants.  First, 
each  participant  was  asked  to  note  (on  a  horizontal  timeline) 
planned  control  actions  that  were  intended  to  solve  the  problem. 
These  actions  were  to  be  predicted  as  far  ahead  as  would  normally 
be  anticipated.  Then,  a  sector  map  was  presented,  anil  participants 
drew  predicted  aircraft  trajectories  as  far  ahead  as  they  could 
determine.  Different  colored  pencils  were  used  to  distinguish  the 
predicted  trajectories  for  each  of  the  seven  aircraft  in  the 
problem. 

In  order  to  discover  how  controllers'  plans  changed  during  problem 
solving,  the  Florence  sector  problem  was  paused  every  3  minutes  (or 
twice)  during  each  experimental  simulation  run.  At  each  juncture, 
participants  were  asked  to  point  out  where  they  were  on  their 
preceding  timeline,  and  to  produce  a  new  timeline  and  map  with  the 
time  of  each  pause  as  a  start  point.  It  was  assumed  that 
participants  would  cycle  through  the  Scan-Project-Plan-Act  stages 
of  decision  making  during  problem  execution,  and  that  stopping  the 
simulation  at  these  points  might  allow  access  to  changes  in  their 
approach  to  the  problem. 

The  Act  part  of  the  decision-making  process  consisted  of  the 
instructions  given  by  participants  to  the  simulated  aircraft  during 
the  simulation  run.  Interactions  with  aircraft  were  video-  and 
audio-taped.  At  the  end  of  each  session,  the  experimenter  played 
back  the  video  tape  of  the  session,  and  asked  the  pazrticipant  to 
describe  the  points  at  which  the  plan  changed,  and  reasons  that 
prompted  such  deviations. 


10  SCTie  complexity  factors  were  not  relevant  given  the  characteristics  of  the 
chosen  sector,  problem,  and  simulation  system. 
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After  data  collection  was  completed,  the  experimenter  reviewed  the 
video  tapes,  and  recorded  all  ATC  actions  taken  by  each 
participant.  The  experimenter  traced  the  actual  flight  paths  of 
the  aircraft  on  a  sector  map  during  video  tape  playback. 

3.4  RESULTS. 

The  results  are  categorized  by  the  decision-making  phase  they 
represented . 

3.4.1  Problem  Analysis  fScan/Proiect) . 

When  asked  to  compare  the  importance  of  the  radar  screen  and 
flight  strips  for  problem  analysis,  every  participant  emphasized 
the  screen  over  the  flight  strips.  The  average  importance  rating 
for  the  screen  was  6.8  (on  a  seven  point  scale).  Participant 
ratings  of  flight  strip  importance  were  lower  and  more  variable 
(averaging  3.7) .  One  controller  commented  that  the  screen  provided 
current  aircraft  status,  while  the  flight  strips  were  useful  for 
information  about  the  future. 

Participants  were  quite  consistent  in  their  indications  of  the 
types  of  information  available  from  each  source.  The  radar  screen 
was  relied  upon  for  aircraft  identifier  (ACID) ,  altitude,  heading, 
and  location  data,  while  flight  strips  provided  aircraft  type  and 
flight  plan.  When  asked  to  evaluate  the  importance  of  each  kind 
of  aircraft  data  for  analyzing  the  Florence  problem,  participants 
consistently  mentioned  altitude,  location,  speed,  and  heading  as 
being  critical.  Flight  plan  and  aircraft  type  were  also 
considered  useful,  but  opinions  were  more  varied.  ACID  was  rated 
least  important  by  most  participants,  with  two  notable  exceptions. 

As  is  illustrated  in  table  16,  controllers  did  not  agree  on  the 
anticipated  number  or  safety  risks  of  potential  conflicts  in  the 
problem.  Participants  projected  an  average  of  six  conflicts 
(ranging  from  five  to  eight) .  In  fact,  when  the  air  traffic 
scenario  was  run  ahead  with  no  controller  intervention,  two  pairs 
of  aircraft  lost  separation  (5  miles  horizontal  or  1000  feet 
vertical) .  Five  participants  predicted  one  of  these  actual 
conflicts,  and  six  predicted  the  second,  assigning  them  a  priori 
probabilities  ranging  from  10  to  100  percent. 
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Table  16.  Predicted  and  Actual  Conflicts 
(Actual  conflicts  in  bold,  n:;sar  conflicts  in  italics) . 
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The  simulated  Florence  problem  had  been  designed  with  a  set  of 
sector  complexity  factors  in  mind.  Each  participant  was  asked  to 
rate  the  problem  on  a  series  of  13  factors,  and  their  ratings  were 
compared  with  those  of  the  Jacksonville  ARTCC  controller  who  had 
designed  the  problem.  The  seven  participants  did  not  always  agree 
on  each  factor,  as  can  be  seen  by  the  ratings  shown  in  table  18. 
To  gauge  agreement  with  the  problem-creator's  ratings,  product- 
moment  correlations  were  calculated  between  all  participants' 
ratings,  and  the  ratings  of  the  controller  who  designed  the 
problem.  Resulting  correlation  coefficients  were  moderate  (r 
values  ranged  between  0.53  and  0.76;  all  but  two  were  significant 
at  p  <  0.05) . 

Before  proceeding  to  describe  their  plans,  the  seven  participants 
were  asked  to  make  three  types  of  ratings.  First,  they  evaluated 
their  familiarity  with  the  Florence  sector  airspace,  then  their 
familiarity  with  the  simulated  Florence  problem,  and  finally  the 
difficulty  of  the  problem  (all  on  seven-point  scales) .  The  average 
rating  for  sector  familiarity  was  3.6;  average  problem  familiarity 
was  5.3,  and  average  problem  difficulty  was  3.8.  However,  these 
averages  did  not  properly  reflect  the  ratings  made  by  this  group, 
given  that  individual  values  varied  considerably  between 
participants.  For  sector  familiarity  and  problem  familiarity, 
there  was  substantial  variability.  Problem  difficulty  ratings  were 
more  consistent. 

3.4.2  Plan. 

The  planning  data  were  of  two  types:  predicted  timelines  and 

predicted  flight  paths.  In  order  to  evaluate  the  timeline 
responses,  each  participant's  anticipated  instructions  were 
compared  with  the  actual  instructions  they  issued  during  the 
simulated  problem.  Several  measures  were  calculated,  as  shown  in 
table  18 . 
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Table  17 .  Sector  Complexity  Ratings  by 
Problem  Designer  and  Participants 
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Table  18,  Measures  of  Planning  Efficiency 
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Number  Predicted  refers  to  the  total  number  of  actions  participants 
said  they  would  make  during  the  three  planning  periods  (initial, 
first  pause,  and  second  pause) .  Number  Actual  was  the  number  of 
instructions  issued  to  aircraft  during  the  simulated  ATC  problem. 
A  D-Match  was  counted  when  there  was  a  direct  pairing  of  a 
predicted  action  with  an  observed  action.  An  Indirect  Match,  or  I- 
Match,  was  counted  when  there  was  sufficiently  close  correlation 
between  predicted  and  actual  to  assume  that  the  planned  event  was 
ultimately  carried  out.  Total  Matches  was  the  sum  of  D  and  I 
matches . 

The  effectiveness  of  each  participant's  predictions  was  calculated 
in  several  ways.  First,  Matches /Actual  (matches  from  pre-scenario 
plan  divided  by  total  number  of  instructions  issued)  reflected  how 
well  the  participant  predicted  events  during  the  entire  problem  at 
the  initial  planning  stage,  before  the  simulation  was  started. 
Matches/Actual  was  the  ratio  of  all  predicted  actions  (including 
initial  planning  and  first  and  second  pauses) ,  divided  by  the  total 
number  of  actions  taken.  This  indicated  participants' 
effectiveness  in  planning  after  gaining  experience  with  the 
problem.  In  some  cases,  they  predicted  an  action  more  than  once. 
This  redundancy  was  estimated  by  dividing  total  matches  by  total 
predicted  (Matches /Predicted) . 

The  planned  flight  paths  drawn  on  the  sector  maps,  prior  to  the 
start  of  the  problem,  were  compared  to  the  actual  paths  followed  by 
the  aircraft.  In  general,  participants  were  able  to  draw  the 
projected  routes  of  aircraft  through  to  the  end  of  the  problem. 
When  comparing  the  planned  and  actual  flight  paths,  it  was  found 
that  the  routes  planned  were  very  similar  to  the  routes  taken, 
except  in  the  case  of  two  aircraft.  In  nearly  every  case,  the 
flight  paths  of  these  aircraft  varied  from  the  plan.  An  exeunple  of 
this  is  shown  in  figure  7. 

Each  participant  was  asked  whether  the  solution  to  the  problem  had 
emerged  fully  formed  (i.e.,  had  been  recognized)  or  had  been 
consciously  worked  out.  Two  participants  illustrated  the  extremes 
in  problem  solving:  One  said  that  the  problem  had  to  be  worked  out 
completely  by  hand,  while  another  indicated  that  the  solution  was 
immediately  self-evident.  However,  the  other  five  participants 
described  the  problem-solving  process  as  being  a  combination  of 
starting  with  self-evident  or  immediately  available  ideas,  and  then 
thinking  them  through  in  terms  of  the  presented  situation.  For 
example,  one  participant  commented  that  initial  steps  to  solve  the 
problem  were  clear,  but  later  steps  had  to  be  worked  out. 
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PLANNED 


ACTUAL 


Figure  7.  Planned  versus  Actual  Flight  Paths 
(Only  deviations  from  plan  shown) 


The  number  of  steps  each  participant  took  to  solve  the  Florence 
problem  varied  from  21  to  32  as  are  shown  in  table  18.  During  the 
pauses  in  the  simulation,  and  at  the  time  of  the  video  tape  review 
of  the  problem,  participants  were  asked  when  and  for  what  reasons 
they  changed  their  plans.  They  reported  making  from  one  to  three 
changes  during  the  entire  problem  run.  An  oft-cited  reason  for  a 
change  in  plan  was  the  unpredictable  behavior  of  one  of  the 
aircraft,  a  military  flight  whose  climb  rate  was  not  known.  Other 
reasons  for  plan  changes  included  detection  of  a  conflict,  a  change 
in  sequencing  method,  for  maintenance  of  spacing,  and  unpredicted 
aircraft  paths. 

3.5  DISCUSSION. 

While  reviewing  the  results,  it  may  be  helpful  to  keep  in  mind  that 
controllers  participating  in  this  experiment  came  from  different 
areas  of  the  Jacksonville  Center,  and  had  varying  amounts  of  ATC 
experience.  This  diversity  may  explain  some  of  the 
inconsistencies . 

3.5.1  Importance  of  Available  Information. 

When  analyzing  the  simulated  Florence  problem,  participants 
stressed  the  importance  of  radar  screen  information.  In 
particular,  they  looked  to  the  screen  for  current  aircraft  status 
including  ACID,  altitude,  heading,  and  location.  Flight  strips 
provided  future  information  regarding  flight  plan,  and  were  also 
relied  upon  for  information  on  aircraft  type.  Participants  were 
not  given  instructions  about  writing  on  the  flight  strips  provided 
to  them  for  the  study,  and  appeared  to  handle  the  simulated  problem 
adequately  without  making  notations. 


Altitude,  location,  heading  and  speed  were  reported  as  the  most 
critical  pieces  of  aircraft  data.  However,  there  was  not 
consistent  agreement  on  the  usefulness  of  other  kinds  of  data,  such 
as  aircraft  type.  One  controller,  for  example,  found  aircraft  type 
information  to  be  highly  critical,  while  others  did  not.  This 
variation  in  emphasis  may  reflect  differences  in  controller  styles. 
Some  controllers  may,  for  example,  use  speed  controls  more 
frequently  in  their  work,  and  this  may  require  more  information  on 
aircraft  performance.  Controller  style  may  develop  as  a  function 
of  on-the-job  training,  i.e.,  a  Developmental  may  develop  a  style 
that  resembles  the  training  controller's  style. 


ACID  was  rated  least  important  by  the  majority  of  participants, 
but  it  was  rated  most  important  by  others.  This  may  have  been 
because  of  different  concepts  about  the  criticality  of  this 
information.  The  primary  function  of  aircraft  flight  number  is  to 
identify  the  target  on  the  radar  screen  and  the  entry  on  the  flight 
strip.  In  a  sense,  it  is  critical  information  in  this  role,  but  it 
provides  very  little  data  about  an  aircraft's  present  or  future 
status.  Depending  upon  one's  point  of  view,  ACID  could  then  be 
either  critical  or  unimportant. 

3.5.2  Conflict  Potential. 

When  evaluating  conflict  potential,  study  participants  were  quite 
conservative,  given  that  only  two  conflicts  would  have  occurred  had 
the  simulated  Florence  problem  been  left  to  run  ahead  with  no 
intervention.  Six  conflicts  were  predicted  by  at  least  five  of  the 
seven  participants,  although  there  was  a  general  lack  of  consensus 
about  the  probability  values  associated  with  each  conflict.  A  few 
participants  did  not  predict  either  of  the  conflicts  that  would 
actually  have  occurred. 

The  variability  inherent  in  the  conflict  data  may  have  several 
sources.  First,  some  of  the  participants  were  not  familiar  with 
the  airspace,  and  this  may  have  made  conflict  prediction  more 
difficult.  It  is  also  not  known  how  far  ahead  controllers 
typically  look  to  identify  possible  problems.  Like  trying  to 
forecast  the  weather  too  far  in  advance,  it  may  have  been 
unrealistic  to  ask  for  conflict  predictions  too  far  ahead  of  a 
normal  "time  window."  Unfortunately,  these  data  do  not  provide  any 
information  on  the  dimensions  of  this  window.  It  is  also  possible 
that  it  was  difficult  for  participants  to  evaluate  conflicts  as  if 
no  intervention  would  occur,  because  controllers  do  not  typically 
perform  such  evaluations. 

Accordingly,  some  of  the  conflict  predictions  may  have  been  made  in 
the  context  of  the  strategy  each  participant  had  developed  for 
managing  the  aircraft,  thus  adding  more  variation  to  the  responses. 
In  any  event,  most  of  the  participants  in  this  experiment  were 
careful  to  identify  any  potential  risks  of  conflict.  Although  most 
of  the  predicted  risks  did  not,  in  reality,  represent  actual 
conflicts  that  would  have  occurred,  such  planning  no  doubt  served 
to  focus  attention  on  potential  problems  until  they  had  been  fully 
evaluated. 
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3.5.3  Effects  Of  Complexity  Factors. 

One  purpose  of  this  study  was  to  consider  the  effects  of  specific 
sector  complexity  factors  on  controller  decision  making.  In  the 
problem  evaluation  stage,  it  was  of  interest  to  discover  whether 
study  participants  and  the  designer  of  the  problem  had  the  same 
perception  of  sector  complexity.  The  designer  of  the  simulated 
Florence  problem  was  a  North  Area  controller.  Participants'  factor 
ratings  correlated  moderately  well  with  the  designer's  evaluation. 
For  the  most  obvious  factors  (Intersecting  Airways,  Multiple 
Functions,  Proportion  of  Departing/Arriving  Traffic,  Airline 
Hubbing,  and  Arrival/Departure  Sequencing  and  Spacing) ,  participant 
ratings  varied.  The  most  consistent  response  was  for  Sequencing 
and  Spacing. 

The  overall  correlations  between  participants  and  the  problem 
designer  indicate  that  most  participants  perceived  the  problem  in 
the  way  it  was  planned.  However,  on  some  of  the  complexity  scales, 
there  was  an  obvious  lack  of  agreement  among  the  participants.  One 
reason  for  this  could  have  been  the  diversity  of  their  backgrounds. 
Only  three  of  the  seven  participants  were  North  Area  controllers. 
Thus,  differences  in  experience  might  have  affected  the  perception 
and  weighting  of  the  sector  complexity  factors.  Another  problem 
could  have  been  the  wording  of  the  statements  in  the  rating  scale. 
Perhaps  participants  did  not  fully  understand  each  complexity 
factor. 

It  also  should  be  recalled  that  sector  complexity  factors  are  not 
objective  qualities  of  the  sector  and  problem.  These  factors  are 
subjectively  interpreted  by  controllers  who  assign  their  own 
importance  levels  to  the  various  factors.  These  importance  levels 
will  affect  how  factors  are  perceived.  The  highest  level  of 
agreement  was  for  Arrival /Devote  nure  Sequencing  and  Spacing,  perhaps 
because  it  was  the  main  focu  of  the  simulated  Florence  problem  and 
was  obvious  to  all  participa  ‘ s. 

Although  controllers  were  not  asked  specifically  to  discuss  their 
actions,  with  regard  to  the  complexity  factors  present  in  the 
problem,  their  planning  and  problem-solving  activities  provided 
information  on  factor  effects.  One  of  the  highest  rated  factors 
was  Multiple  Functions  (see  table  16) .  Participants  had  to  handle 
arrivals,  a  departure,  and  an  overflight.  No  specific  problem¬ 
solving  actions  can  be  directly  connected  to  this  factor  in  that 
the  plans  made  and  executed  could  be  related  to  more  than  one 
requirement,  as  discussed  below. 
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The  next  three  most  predominant  factors  in  the  problem  were  related 
to  traffic  flow  to  and  from  airports.  (Five  out  of  the  total  of 
seven  aircraft  were  flying  into  the  Chesterfield  airport,  and  there 
was  one  military  departure.)  Thus,  complexity  factors  in  the 
problem  included  Airline  Hubbing,  Arrival /Departure  Sequencing  and 
Spacing,  and  Number  of  Departing  or  Arriving  Aircraft  (see  table 
16) .  Traffic  Volume  also  played  a  role  in  terms  of  the 
concentration  of  aircraft  along  one  route. 

To  summarize,  the  highest  rated  complexity  factors  were  all  related 
to  the  theme  of  arrivals  and  departures  concentrated  in  one  area. 
Nhen  participants  planned  their  solutions  to  the  simulated  problem, 
their  major  focus  was  on  how  to  route  all  five  of  the  arriving 
Chesterfield  flights  so  that  correct  spacing  could  be  created  and 
maintained.  The  effect  on  behavior  was  that  aircraft  were  moved 
away  from  their  original  flight  plans  in  order  to  achieve  this 
goal,  and  to  reduce  the  conflict  potential  posed  by  the  number  of 
intersecting  airways,  another  important  complexity  factor  (see 
figure  7) , 

Sector  size  probably  limited  the  number  of  complex  routings  that 
could  be  considered.  The  presence  of  overflying  aircraft  provided 
another  reason  for  maneuvering  the  arriving  aircraft  away  from 
their  flight  plans  in  order  to  avoid  conflicts.  A  departing 
military  flight  played  a  significant  role  in  causing  participants 
to  deviate  from  their  initial  plans;  its  climb  rate  was  potentially 
much  faster  than  the  commercial  aircraft  and  had  to  be  assessed  as 
the  problem  progressed. 

3.5.4  Sector  and  Problem  Katinas. 

Participant  judgments  were  indicative  of  the  differences  in 
familiarity  with  the  simulated  Florence  sector  and  problem.  As 
might  be  expected,  the  three  participants  from  the  North  Area  were 
consistent  in  evaluating  the  sector  and  problem  as  familiar.  Other 
participants  gave  lower  ratings.  This  is  understandable 
considering  that  they  lacked  experience  with  the  North  Area 
airspace.  Participant  evaluations  of  problem  difficulty  were  more 
consistent,  averaging  3.8  (on  a  seven  point  scale).  Perceptions  of 
difficulty  and  familiarity  were  not  significantly  correlated  with 
years  of  experience. 

_ Eisoning- 

Air  traffic  controllers  are  encouraged,  during  their  training,  to 
form  a  plan  for  handling  air  traffic  situations.  Given  the  short 
duration  of  the  simulated  problem  employed  for  this  study,  the 
first  planning  session  (immediately  preceding  the  start  of  the 
problem)  was  chosen  to  be  the  focus  of  this  discussion. 
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The  number  of  predicted  instructions  during  the  first  planning 
period  varied  from  13  to  37.  Some  participants  (judging  by  their 
vocalizations  during  the  planning  process)  painstakingly  planned 
all  anticipated  actions,  while  others  plotted  their  initial  moves 
and  then  adopted  a  "wait  and  see”  approach  to  further  actions. 
Although  differences  between  participants  in  the  number  of  planned 
actions  may  be  related  to  ability,  variations  also  may  be 
attributable  to  type  of  decision-making  style  or  to  other 
variables. 

Niimber  of  predicted  instructions  had  a  moderate  negative,  but  non¬ 
significant  correlation  with  years  of  ATC  experience  (r  «  -.52). 
This  suggests  (inconclusively)  that  increasing  ATC  experience  may 
be  related  to  a  reduction  in  conscious  planning  activities.  This 
could  be  expected  given  the  assumption  that  skills  become  more 
automatic  and  less  accessible  to  consciousness  with  extended 
practice. 

Number  of  actual  instructions  was  much  less  varied  than  predicted 
Instructions.  The  average  number  of  actual  instructions  was  27. 
This  suggests  that,  although  participants  used  somewhat  different 
strategies  to  solve  the  simulated  problem,  the  number  of 
instructions  they  actually  issued  was  similar.  The  number  of  steps 
taken  appears  to  be  less  dependent  on  between-partlcipant 
differences.  A  moderate,  but  non-significant  relationship  (r  « 
.63)  was  found  between  years  of  experience  and  number  of  actual 
instructions.  There  was  a  significant  correlation  between  actual 
instructions  and  problem  difficulty  rating  (r  •  .78,  p  <  .05). 
This  finding  suggests  that  perception  of  problem  difficulty  may  be 
positively  related  to  the  number  of  steps  actually  taken  (as 
opposed  to  predicted)  to  solve  it. 

The  percentage  of  first  matches /actual  indicated  how  well  each 
participant's  initial  plan  matched  actions  taken  during  the  problem 
scenario.  Effectiveness  varied  from  16  percent  to  73  percent. 
There  was  no  significant  correlation  with  years  of  experience,  (r 
*  -.52,  p  >  .05),  although  the  size  of  the  correlation  suggests 
that,  given  more  data,  controllers  with  less  experience  (or  who  are 
yovinger)  might  prove  to  be  more  successful  planners.  However,  as 
noted  above,  a  style  that  is  not  supportive  of  conscious  planning 
would  also  reduce  predictive  effectiveness.  Degree  of  planning 
eddility  was  not  related  to  specialization  area;  North  controllers 
were  not  any  more  or  less  effective  than  were  controllers  from 
other  areas. 
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Although  there  was  no  quantitative  method  available  to  assess  the 
accuracy  of  predicted  aircraft  routes  drawn  by  each  participant,  it 
was  possible  to  make  visual  comparisons.  All  participants  drew 
aircraft  routes  for  most  of  the  anticipated  actions.  It  appeared 
that  it  was  easier  for  some  controllers  to  construct  a  visual  plan 
in  two  dimensions,  as  opposed  to  developing  a  verbal  list  of 
anticipated  instructions.  In  most  cases,  actual  aircraft  routes 
did  not  deviate  significantly  from  planned  flight  paths,  except  in 
the  case  of  two  aircraft.  These  aircraft,  along  with  five  others, 
had  flight  plans  which  had  them  crossing  over  the  same  fix  within 
a  few  minutes  of  each  other.  For  some  reason,  these  two  aircraft 
were  given  ATC  instructions  to  deviate  from  their  planned  routes, 
perhaps  due  to  unexpected  problems  with  negotiating  the  other 
traffic  past  the  common  fix. 


Probing  for  RPD  was  limited  to  a  question  to  each  participant 
regarding  how  the  problem  solution  was  developed.  Judging  by  their 
reports,  five  of  the  seven  participants  indicated  that  a  basic 
approach  was  immediately  self-evident.  One  participant  said  that 
the  solution  had  emerged  fully  formed.  From  these  statements,  we 
can  assume  that  at  least  some  part  of  the  problem  solution  was 
recognized,  rather  than  developed  from  first  principles. 

Even  those  participants  who  rated  the  problem  as  highly  familiar, 
however,  indicated  that  some  fine  tuning  was  required  to  complete 
the  solution.  Nevertheless,  this  is  in  line  with  Klein's  (1989) 
description  of  the  RPD  process.  The  participant  who  reported 
consciously  working  the  problem  out,  also  indicated  the  lowest 
degree  of  familiarity  with  the  sector  and  problem  (one  on  scale  of 
seven) .  The  other  six  participants  averaged  six  out  of  seven  when 
rating  problem  familiarity.  This  pattern  of  results  shows  some 
support  for  RPD.  RPD  requires  familiarity  and  experience  with  the 
problem  domain  and  the  problem  type.  Someone  who  is  unfamiliar  and 
inexperienced  cannot  be  expected  to  recognize  the  problem  or  its 
solution. 

During  the  enactment  of  the  problem,  participants  changed  their 
initial  plans  from  one  to  three  times.  Reasons  for  changes 
appeared  to  fall  within  three  categories:  unanticipated  pilot 
actions,  inaccurate  projections  of  aircraft  movement,  and  actions 
that  were  not  anticipated  in  the  initial  plan.  The  number  of 
changes  made  by  a  given  participant  did  not  correspond  with  the 
effectiveness  of  their  initial  plan. 
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3.6  PHASE  II  CONCLUSIONS. 

The  results  of  this  preliminary  decision-maXing  study  indicate  that 
en  route  air  traffic  controllers  are  much  more  reliant  on 
information  from  the  radar  screen  as  opposed  to  flight  strip 
information,  with  aircraft  altitude,  location,  heading,  and  speed 
being  the  most  critical  data.  While  the  screen  offers  data  on 
current  status,  flight  strips  are  useful  for  future  information. 
In  the  limited  environment  of  the  simulated  ATC  problem,  it 
appeared  that  paper  strips  were  not  needed  as  a  mnemonic  aid  and 
could  be  dispensed  with  altogether,  given  that  flight  plan  and 
aircraft  type  information  could  be  posted  on  the  screen.  However, 
this  issue  requires  more  study  because  only  a  limited  form  of 
electronic  strip  was  visible  on  the  ATC  simulator  interface. 

The  controllers  in  this  study  were  not  consistent  in  evaluating 
conflict  potential,  perhaps  for  reasons  inherent  in  the  experiment. 
However,  they  generally  adopted  a  conservative  approach  by 
identifying  more  problems  than  would  have  actually  occurred.  This 
cautious  attitude  has  been  identified  in  another  study  (Bisseret, 
1981)  and  is  a  prudent  one  to  adopt  in  the  ATC  environment. 

Participants'  cognizance  of  the  complexities  inherent  in  the  sector 
showed  some  similarity  to  those  the  problem  designer  had  in  mind. 
Differences  may  have  been  due  to  varying  levels  of  familiarity  with 
the  problem  and  sector.  It  was  possible  to  infer  that  some  of  the 
primary  complexity  factors,  in  the  sector  and  problem,  influenced 
controller  behavior. 

There  was  ample  evidence  that  the  complexity  factors  present  in  the 
problem  affected  problem-solving  behavior.  However,  it  was  not 
always  possible  to  determine  exactly  which  factor  a  given  action 
was  addressing.  More  detailed  verbal  reports  by  participants 
during  viewing  of  problem  video  tapes  would  have  been  helpful  in 
identifying  specific  effects. 

Effectiveness  in  problem  planning  varied  between  participants  as 
did  the  accuracy  of  their  first  plan.  One  participant  was  able  to 
account  for  73  percent  of  actual  instructions  before  starting  the 
problem.  Differences  in  planning  may  have  been  due  to  several 
factors,  including  amount  of  experience  and  controller  style.  With 
more  ATC  background,  conscious  planning  may  give  way  to  a  "wait  and 
see"  attitude  that  comes  from  confidence  in  we  11 -developed  problem¬ 
solving  skills.  More  data  are  recjuired  to  explore  this  topic.  The 
nvimber  of  steps  taken  to  solve  the  problem  increased  with 
perception  of  problem  difficulty. 
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It  appeared  that  controller  participants  had  less  difficulty  with 
reporting  their  plans  in  a  visual  format  (as  opposed  to  vnriting 
anticipated  instructions  on  a  time  line) .  This  may  support  the 
contention  that  cognitive  activities  processed  in  a  primarily 
visual-spatial  modality  should  be  researched  using  the  same  format; 
information  may  be  lost  when  switching  to  a  verbal  output. 
However,  it  also  should  be  noted  that  simpler  (two  dimensional) 
information  was  required  to  complete  the  maps  (altitude  and  speed 
changes  were  omitted) . 

There  was  evidence  for  RPD  in  that  those  participants  reporting 
familiarity  with  the  problem  indicated  that  at  least  part  of  the 
solution  emerged  without  conscious  deliberation.  This  finding 
would  not  be  particularly  surprising  if  all  the  participants  were 
from  the  same  area,  and  had  the  same  level  of  familiarity  with  the 
specific  problem.  However,  the  participants  were  from  different 
areas,  and  were  apparently  able  to  transfer  their  knowledge 
automatically.  This  finding  suggests  that  analogous  problems  occur 
in  other  specialization  areas,  which  is  a  reasonable  possibility. 
For  example,  a  separation  problem  in  one  sector  may  be  similar  at 
some  level  to  a  separation  problem  in  another  sector;  therefore, 
the  solution  is  probably  similar  at  some  level.  Thus,  one  can 
expect  a  certain  level  of  ATC  expertise  and  RPD  to  transfer  between 
areas.  The  main  feature  of  RPD,  that  alternatives  are  not  laid  out 
and  selected  in  a  parallel,  deliberative  fashion,  but  emerge  as  if 
by  recognition  in  a  serial  process,  seems  to  have  been  in  evidence. 

Indications  that  participants  cycle  through  the  hypothesized 
decision-making  process  (as  shown  in  figure  6)  a  number  of  times, 
even  during  a  short-term  problem  situation,  is  shown  by  their 
reports  that  they  changed  their  plans  up  to  three  times.  Reasons 
for  these  changes  included  unanticipated  pilot  actions,  inaccurate 
projections  of  aircraft  movement,  and  actions  that  were  not 
anticipated  in  the  initial  plan. 

Further  research  is  needed  to  validate  the  findings  of  this 
preliminary  study  and  to  extend  the  investigation  of  controller 
decision  making.  Additional  techniques  should  be  developed  to  link 
decision-making  behavior  to  cognitive  structures  (i.e.,  sector 
complexity  factors) . 

4.  CONCLUSIONS. 

The  focus  of  Phase  I  of  the  reported  studies  was  to  examine 
controller  cognitive  performance  by  considering  the  environment  in 
which  the  controller  works.  It  was  thought  that  if  a  clearer 
understanding  of  the  variables  that  contribute  to  sector  complexity 
was  established,  it  would  facilitate  research  on  controller 
cognition  and  decision  making. 
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To  this  end,  two  main  approaches  (direct  and  indirect)  were  used  to 
investigate  sector  complexity  in  the  Jacksonville  ARTCC.  This 
resulted  in  the  development  of  a  set  of  19  candidate  complexity 
factors,  11  of  which  correlated  with  controller  judgments  of 
overall  sector  complexity.  The  19  factors  were  fuxrther  refined 
down  to  a  set  of  15.  These  factors  should  prove  useful  in  future 
research  efforts  where  the  working  environment  of  the  controller 
must  be  considered  or  manipulated  in  some  way.  However,  it  should 
be  noted  that  these  factors  were  developed  at  a  specific  Federal 
Aviation  Administration  (FAA)  site  and  sx>  generalizations  to  other 
air  traffic  facilities  should  be  made  with  caution. 

Given  the  establishment  of  these  complexity  factors,  it  was  then 
possible  to  complete  a  preliminary  study  of  controller  decision 
making  in  Phase  II.  An  experiment  was  conducted  at  Jacksonville 
ARTCC  using  an  Air  Traffic  Control  (ATC)  simulator  to  gather 
various  kinds  of  data  on  controller  problem  solving  behaviors.  As 
part  of  this  study,  relationships  were  drawn  between  the  complexity 
factors  evident  in  the  chosen  problem  and  controller  actions.  It 
became  evident  that,  in  studies  of  sector  complexity  and  controller 
decision  making,  it  is  not  possible  to  separate  sector 
characteristics  from  air  traffic  patterns.  These  aspects  of  the 
controller's  environment  interact  to  produce  the  various  types  of 
complexity  observed  in  Phase  I. 

Although  interesting  data  were  collected  as  part  of  Phase  II,  it 
was  evident  that  further  work  should  be  completed  to  relate 
complexity  factors  to  controller  behaviors.  It  is  not  sufficient 
to  know  that  certain  complexity  factors  exist  without  investigating 
their  effects  on  controller  decision  making,  workload,  and 
performance.  It  is  recommended  that  the  studies  reported  in  this 
document  be  used  as  an  input  for  future  research  in  this  area. 
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APPENDIX  A 


BRIEF  INTRODUCTION  TO  MULTIDIMENSIONAL 
SCALING  AND  PATHFINDER  ANALYSIS 


MULTIDIMENSIONAL  SCALING  (MDS) . 

MDS  offers  a  systematic  way  to  measure  and  understand  the 
relationships  between  objects  when  the  underlying  dimensions  that 
make  them  similar  or  different  are  not  known.  MDS  is  a  "scaling 
technique"  that  uses  direct  similarity  judgments  as  input.  These 
judgments  are  made  by  participants  who  use  a  rating  scale  to 
indicate  the  degree  of  similarity  between  the  concepts  or  objects 
of  interest.  Those  making  these  judgments  do  not  need  a  conscious 
knowledge  of  the  attributes  of  the  objects  they  are  comparing 
(i.e.,  what  makes  them  similar  or  different).  Their  ratings 
indirectly  reflect  their  knowledge  about  the  stimuli,  and  MDS  can 
display  the  results  in  a  spatial  map. 

MDS  processes  the  numbers  generated  from  people's  judgments,  and 
prints  out  a  map  (such  as  shown  in  figure  Al)  that  represents 
objects  judged  as  similar  to  each  other  as  points  close  together  in 
space.  Objects  judged  as  dissimilar  are  represented  as  points 
distant  from  one  another.  The  objects*  interrelationships  may  best 
be  explained  by  only  a  single  dimension.  However,  two  or  more 
dimensions  are  often  generated.  The  MDS  process  indicates  which 
nxmber  of  dimensions  fits  the  data  best  by  creating  various 
statistical  measures.  Once  the  MDS  "space"  has  been  built,  it  is 
then  up  to  the  experimenter,  with  help  from  the  original  judges  or 
other  SMEs,  to  determine  why  certain  objects  or  concepts  are  seen 
as  similar,  and  why  others  are  perceived  to  be  different. 

Figure  Al  shows  a  possible  MDS  solution  for  similarity  judgments 
between  aviation  concepts.  Each  dimension  of  the  MDS  graph 
represents  one  possible  underlying  factor  that  defines 
relationships  between  the  concepts.  In  this  example,  the 
horizontal  dimension  could  be  related  to  an  air  traffic  (ground) 
versus  aircraft  (airborne)  factor,  and  might  be  called  "location." 
The  vertical  dimension  could  be  related  to  quantities  (altitude)  as 
opposed  to  equipment,  and  might  be  named  "concreteness."  As  can  be 
seen,  a  certain  amount  of  subjective  judgment  enters  the  equation 
to  produce  the  final  result.  However,  if  SMEs  are  used  to  aid  in 
this  process,  valuable  insights  can  be  gained  into  the 
relationships  between  concepts  or  objects. 

PATHFINDER  ANALYSIS. 

Pathfinder  is  a  mathematical  procedure  that  uses  the  same  "distance 
data"  employed  by  MDS,  but  generates  a  different  spatial  map  of  the 
data.  Distance  data  are  usually  similarity  ratings,  which  imply 
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Figure  Al.  A  Hypothetical  MDS  Result  for  Aviation  Concepts 
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the  "psychological"  distance  or  similarity  between  concepts  and 
terms.  A  Pathfinder  network  consists  of  a  set  of  concepts  and 
links  that  directly  connect  pairs  of  concepts  that  are  highly 
related.  If  Pathfinder  has  tapped  the  organization  of  the 
participant's  knowledge,  the  resulting  graph  should  represent  the 
basic  organization  of  existing  elements,  based  on  an  individual's 
experience . 

For  example,  a  Pathfinder  network  could  be  derived  from  the  same 
set  of  similarity  judgments  about  aviation  concepts.  A  network 
that  displayed  the  result  might  look  like  figure  A2.  Each 
concept's  relationship  to  each  other  is  displayed.  If  the 
commonalities  between  concepts  are  not  known,  a  Pathfinder  network 
can  display  them.  However,  as  with  MDS,  it  is  then  up  to  the  user 
to  define  the  nature  of  the  links  between  the  concepts.  While  MDS 
tends  to  offer  general  information  about  a  set  of  concepts. 
Pathfinder  can  give  a  more  detailed  look  at  their 
interrelationships.  For  example,  figure  A2  shows  three  clusters  of 
concepts:  aircraft  parameters  (altitude,  heading,  and  speed) , 
airport  concepts  (runway,  ATC,  radar,  and  tower) ,  and  aircraft 
parts  (flaps,  wings,  engine) .  The  length  of  the  link  lines  between 
the  concepts  demonstrates  the  degree  of  relationship  (shorter  is 
more  similar  or  more  closely  related) . 

MDS  and  Pathfinder  offer  mathematically-based  methods  to  look  for 
hidden  or  unknown  relationships  between  a  large  set  of  objects. 
These  techniques  can  help  uncover  information  that  might  not  be 
immediately  obvious  to  those  familiar  with  the  subject  matter. 


Figure  A2.  A  Hypothetical  Pathfinder  Network 
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MDS  RESULTS  AND  INTERPRETATIONS  FOR 
SPECIALIZATION  AREAS  (MAIN  STUDY) 

NORTH  SPECIALIZATION  AREA. 

The  Kendall  correlation  for  the  North  Area  complexity  similarity 
judgments  was  x^{35/  N  =  5)  =  101.42,  p  =  .0000,  indicating  a 

significant  level  of  agreement  between  FPLs  about  which  pairs  of 
sectors  were  similar  in  complexity. 

Stress  for  the  overall  two  dimensional  solution  was  0.20,  and 
amount  of  variance  accounted  for  in  the  original  proximity  matrix 
was  0.66.  Stress  and  for  the  five  participants  were: 

Participant  Sfecgsg 


1 

0.20 

0.66 

2 

0.20 

0.67 

3 

0.21 

0.64 

4 

0.22 

0.62 

5 

0.19 

0.71 

The  shared  stimulus  space  accounted  for  a  moderate  euaount  of 
variance  for  all  participants.  The  stimulus  space  for  the  five 
North  controllers  is  shown  in  figure  Bl.  Dimension  one  related  to 
radio  frequency  congestion,  and  dimension  two  organized  sectors  by 
how  difficult  they  were  to  manage  as  traffic  density  increased. 
All  five  of  the  participants  emphasized  the  two  dimensions  about 
equally  (see  flgxire  B2)  . 

SOUTH  SPECIALIZATION  AREA. 

The  Kendall  correlation  for  the  South  Area  complexity  similarity 
judgments  was  x*(20,  N  *  5)  =  47.95,  p  *  .0004,  indicating  a 
significant  level  of  agreement  about  which  pairs  of  sectors  were 
similar  in  complexity. 

Stress  for  the  overall  two  dimensional  solution  was  0.20,  and 
amount  of  variance  accounted  for  in  the  original  proximity  matrix 
was  0.75.  Obtained  stress  and  R^  values  were  as  follows: 


Participant 

gtr.g?.g 

1 

0.25 

0.65 

2 

0.05 

0.99 

3 

0.24 

0.63 

4 

0.20 

0.76 

5 

0.20 

0.73 
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DERIVED  STIMULUS  CONFIGURATION: 
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FIGURE  Bl.  MDS  STIMULUS  SPACE  FOR  THE  NORTH 
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DERIVED  PARTICIPANT  WEIGHTS: 

DIMENSION  1  (HORIZONTAL)  VS  DIMENSION  2  (VERTICAL) 
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As  shown  in  figure  B3,  the  shared  stimulus  space  accounted  for  a 
moderate  to  high  amount  of  variance  for  individual  participants. 
Dimension  one  had  to  do  with  volume  and  flow  of  traffic 
(arrivals/departures  versus  overflights) ,  emd  dimension  two  was 
defined  by  whether  a  sector  mainly  had  non-professional  pilots 
lying  under  Visual  Flight  Rules  (VFR)  or  professional  pilots  flying 
under  Instrument  Flight  Rules  (IFR) .  The  weight  space  (figure  B4) 
for  the  South  controllers  reflected  diversity  in  the  use  of  the  two 
dimensions.  Two  participants  emphasized  the  VFR  vs.  IFR  dimension, 
while  the  other  three  used  the  traffic  volume/flow  dimension,  with 
participant  2  placing  exclusive  emphasis  on  this  dimension. 


EA.ST..5£EglALlZATIQN  AREA- 


The  Kendall  correlation  for  the  East  Area  controllers'  complexity 
similarity  judgments  was  x^(27,  N  =  5)  =  38.99,  p  =  .0635, 
indicating  only  a  marginal  level  of  agreement  about  which  pairs  of 
sectors  were  similar  in  complexity.  Stress  for  the  two  dimensional 
solution  was  0.22,  and  amount  of  variance  accounted  for  in  the 


original  proximity 
obtained  for  stress 

matrix 
and  R^: 

was  0.68 

Participant 

Stress 

1 

0.23 

0.60 

2 

0.21 

0.73 

3 

0.28 

0.50 

4 

0.13 

0.92 

5 

0.25 

0.65 

The  following  values  were 


The  shared  stimulus  space  accounted  for  a  moderate  eunount  of 
variance  for  individual  participants.  Participant  3's  judgments, 
however,  were  not  very  well  represented. 


The  stimulus  space  for  the  five  East  controllers  Is  shown  in  figure 
B5.  Interpretation  of  the  dimensions  was  difficult,  even  with  the 
assistance  of  the  Jacksonville  ARTCC  controllers.  The  only 
dimension  that  emerged  was  diagonal,  and  related  to  sectors  with 
one-way,  as  opposed  to  crossing,  traffic. 


The  weight  space  (figure  B6)  was  difficult  to  interpret  given  that 
it  was  not  possible  to  name  the  vertical  and  horizontal  dimensions 
in  the  stimulus  space.  The  graph  shows,  however,  that  only  one 
participant  used  both  dimensions  equally,  while  the  others  placed 
more  emphasis  on  dimension  one  or  two. 

WEST  SPECIALIZATION  AREA. 


The  Kendall  correlation  for  the  West  Area  complexity  similarity 
judgments  was  x^(20,  N  =  5)  =  36.16,  p  =  .0147,  indicating  a 

significant  level  of  agreement  about  which  pairs  of  sectors  were 
similar  in  complexity. 
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2.1  -+ 


1.0  -+ 


0.0  -+- 


GAINSVL 

TAYL 

SILSPGS 


ST. AUG 


-1.0 


-2.1  -+ 


GRMCOV 

ST.JOHNS 


REYSTN 


HI9A1  Volume 
*  Arrival /Departure 


Lov  Volune 
Overfllg^s 


-2.5  -1.5 


-0.5 


0.5 


1.5 


2.5 


VFR 

MO  Prof.  Pilots 


No  VFR 
Prof.  Pilots 


FIGURE  B3.  MDS  STIMULUS  SPACE  FOR  THE  SOUTH 
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OERTVEO  STIMULUS  CQNFIGUBATXC»I: 
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FIGURE  B6.  PARTICIPANT  WEIGHTS  FOR  EAST 
SPECIALIZATION  AREA. 


Stress  for  the  two-dimensional  solution  was  0.21,  and  the  amount  of 
variance  accounted  for  in  the  original  proximity  matrix  was  0.67. 
Stress  and  values  were  as  follows: 


Stress 

1  .25  .66 

2  .30  .24 

3  .10  .96 

4  .18  .76 

5  .17  .76 


The  stimulus  space  accounted  for  a  moderate  to  high  degree  of 
variance  for  all  but  one  of  the  West  FPLs.  Participant  2's  data 
obtained  a  low  value,  indicating  that  this  person's  judgments  did 
not  correspond  well  with  the  others. 

The  stimulus  space  for  the  five  West  controllers  is  shown  in  figure 
B7.  Dimension  one  represented  volume  of  traffic  and  number  of 
military  flights.  Dimension  two  was  oriented  aroimd  the  level  of 
traffic  (altitude,  overflights  versus  arrivals/departures)  and  the 
presence  of  VFR  traffic. 

Weight-space  results  (figiire  B8)  showed  a  considerable  degree  of 
variability  between  participants  in  their  emphasis  on  each  of  the 
two  dimensions.  Participants  l  and  3  based  their  similarity 
ratings  on  traffic  volume,  including  military  traffic. 
Participants  2  and  5  based  their  ratings  on  traffic  level  and 
presence  of  VFR  aircraft.  Participant  4  used  both  dimensions  when 
medcing  similarity  judgments. 

central  gPSglALIZATIPy  APSA. 

The  Kendall  correlation  for  the  Central  Area  complexity  similarity 
judgments  was  x^(20,  N  =  5}  =  41.58,  p  =  0.0031,  indicating  a 
significant  level  of  agreement  about  which  pairs  of  sectors  were 
similar  in  complexity. 

Stress  for  the  two-dimensional  solution  was  0.23,  and  the  amount  of 
variance  accounted  for  in  the  original  proximity  matrix  was  0.73. 
Stress  and  R^  values  were  as  follows: 


Stress 

1 

0.14 

0.86 

2 

0.29 

0.65 

3 

0.28 

0.54 

4 

0.22 

0.85 

5 

0.18 

0.77 
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DERIVED  STIMULUS  CONFIGURATION: 
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FIGURE  B7.  MDS  STIMULUS  SPACE  FOR  THE  WEST 
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shared  stimulus  space  accounted  for  a  moderate  to  high  amount 
of  variance  for  individual  participants.  The  stimulus  space  for 
the  five  Central  controllers  is  sho%m  in  figure  B9.  Dimension  one 
vas  related  to  the  amount  of  coordination  and  number  of  functions 
required  in  a  sector.  Dimension  two  separated  sectors  with  a  large 
proportion  of  overflights  as  opposed  to  climbing  and  descending 
traffic. 

Two  participants  used  both  dimensions  about  equally  in  their 
judgments  (figure  BIO).  Participants  2  and  4,  however,  were  more 
extreme  in  their  use  of  the  dimensions,  respectively  emphasizing 
coordination  or  type  of  traffic  almost  exclusively. 


0  j  ^  CO  ;  M ; » ^  s.-* f f  e  .v  4  03  a 


The  Kendall  correlation  for  the  complexity  similarity  judgments 
made  by  Developments Is  (i.e.,  controller  trainees)  from  the  East 
Area  was  N  •  S)  «■  75.55,  p  •  .0000,  indicating  a  significant 
level  of  agreement  about  which  pairs  of  sectors  were  similar  in 
complexity. 

Stress  for  the  two-dimensional  solution  was  0.23,  and  the  amount  of 
variance  accounted  for  in  the  original  proximity  matrix  was  .78. 
Stress  and  values  were  as  follows: 


PirticiPdnt 

auziBi 

1 

0.41 

0.29 

2 

0.16 

0.91 

3 

0.06 

0.98 

4 

0.25 

0.73 

5 

0.06 

0.99 

The  shared  stimulus  space  accounted  for  a  low  to  high  amount  of 
variance  for  individual  participants  (figure  Bll) .  Participant  I's 
responses  were  not  well  represented.  Dimension  one  described 
sectors  in  terms  of  their  east-west  orientation  and  direction  of 
traffic,  while  dimension  two  was  concerned  with  size  of  airspace 
and  sector  boundaries. 

The  weight  space  for  this  group  show  (figure  D12)  some  divergence 
in  participants*  use  of  the  dimensions.  Three  participants 
weighted  the  two  dimensions  about  equally,  while  two  others  placed 
most  emphasis  on  only  the  first  or  second  dimension.  There  were 
some  similarities  in  the  knowledge  structures  of  East  controllers 
and  Bast  Developments  Is.  Some  pairs  of  sectors  were  viewed  as 
nearly  identical  by  both  groups. 
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FIGURE  Bll.  IDS  STIMULUS  SPACE  FOR  DEVELOPMBNTALS 
(BAST  SPECIALIZATION  AREA) 
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F1GUK£  B12.  PARTICIPANT  WEIGBTS  FOR  DEVELOPMENTALS 
(EAST  SPECIALIZATION  AREA)  . 
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AIRSPACE  PROCEDURES  SPECIALISTS  fAPSs)  AND  TRAFFI<LMANAgEMEHT  .UNIT 

IDfia- 

The  best  fitting  MDS  space  for  the  APSs  was  two  dimensional,  with 
average  stress  -0.19  and  »  0.84.  INDSCAL's  common  stimulus  and 
weight  spaces  for  the  APSs  is  shown  in  figures  B13  and  B14. 
Obtained  values  for  stress  and  R^  were  as  follows  for  Airspace 
Procedure  participants: 

PflrtigiBftnt  fi’ 

1  0.14  0.91 

2  0.20  0.85 

3  0.21  0.76 

Only  one  participant  used  both  dimensions  equally,  while  the  others 
placed  more  emphasis  on  dimension  one  or  two. 

For  the  THU  staff,  the  MDS  space  was  three  dimensional  with  average 
stress  *0.19  and  R^  *  0.80  (see  figures  B15,  through  B20) .  Values 
for  the  individual  THU  participants  are  shown  below: 

Participant  SSXMMM 

1  0.20  0.78 

2  0.19  0.60 

3  0.19  0.85 

Participant  l  primarily  used  dimension  two;  participant  2  used  all 
three  dimensions  approximately  equally  (and  had  the  lowest 
weirdness) ;  and  participant  3  stressed  dimension  one. 

The  larger  number  of  sectors  in  the  Airspace  ProcedurM  and  TMU 
data  sets  supported  the  use  of  a  statistical  approach  to  assist 
with  the  interpretation  of  these  stimulus  spacM.  There  was  also 
an  interest  in  contrasting  such  a  procedure  with  the  more 
subjective  method  used  with  the  FPL  data.  The  method  in  question 
involved  mapping  known  quantities  (in  this  case,  the  previously 
gathered  ratings  of  sector  characteristics)  or  "property  vectors" 
onto  sach  space  using  multiple  regression  (Kruskal  and  Wish,  1988) . 
This  would  result  in  a  set  of  complexity  factors,  similar  to  that 
derived  for  the  specialisation  area  MDS  data  using  manual  methods. 

The  property  vectors  generated,  using  this  technique,  are  similar 
to  the  dimensions  created  by  the  MDS  analysis,  and  plotted  on  the 
ms  graphs.  However,  they  may  not  be  identical  to  them  and  may  be 
at  different  angles.  They  are  plotted  to  result  in  ths  most 
efficient  statistical  "fit"  to  the  data. 
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DERIVED  STIMULUS  CXMFIGURRTIOM: 

DDAOeiCM  1  (HOAIZONIAL)  VS  DIMENSION  2  (VERTICAL) 
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OCStIVED  STXHUtUS  COHTlGUItXTIOH: 

oiMEMSiOK  L  (HOiazcama.)  vs  odcmsion  2  (Verticai.i 
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DERIVED  STIMULUS  CONFIGURATION: 

OZMEMSIC»l  1  (HORIZONTAL)  VS  DIMENSION  3  (VERTICAL) 
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DEBZVED  STIMULUS  CONFIGUIUVTIOM: 
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FIGURE  B17.  TMU  DIMENSIONS,  TWO  VERSUS  THREE 
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FIGURE  B18.  PARTICIPANT  NEIGBTS  FOR  TMU  STAFF, 
ONE  VERSOS  TWO. 
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DEitxvn)  PMcnciipAtir  MEicars: 
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riGURC  B19.  PARTICIPANT  NEIGRTS  FOR  TMO  STAFF,  ONE 

VERSUS  THREE. 
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DERIVED  PARnCIPANT  NEIGBTS: 

DIMENSION  2  (HORIZONTAL)  VS  DIMENSION  3  (VERTICAL) 


FIGURE  B20.  PARTICIPANT  WEIGHTS  FOR  TMD  STAFF,  TWO 

VERSUS  THREE. 


In  order  to  locate  property  vectors,  ratings  of  sectors  on  the 
initial  complexity  factors  were  regressed  over  the  set  of  MDS 
stimulus  coordinates  for  the  dimensions  for  each  group  (APSs  or 
TMU)  .  Those  factors  that  had  significant  multiple  correlations 
with  the  MDS  dimensions  were  assumed  partially  to  explain  the 
configuration  of  the  sectors  in  the  stimulus  space.  Ratings  of  all 
38  Jacksonville  ARTCC  sectors  on  the  24  initial  complexity  factors 
were  used  as  dependent  variables  in  an  attempt  to  identify  property 
vectors.  In  each  case,  it  was  possible  to  fit  property  vectors 
that  accounted  for  an  acceptable  amount  of  variance,  that  were 
statistically  significant,  and  that  were  correlated  with  one  or 
more  of  the  dimensions  in  the  stimulus  space  (Kruskal  and  Wish, 
1988)  . 

Table  B1  shows  the  property  vectors  for  the  APSs.  The  rightmost 
column  shows  the  regression  beta  weights  and  their  correspondence 
with  MDS  dimensions.  The  sign  of  each  beta  weight  indicates  the 
angle  of  the  property  vector  in  the  MDS  stimulus  space. 

In  each  case,  for  the  APS  data,  more  than  one  property  vector  was 
correlated  with  each  dimension  (table  Bl) .  Given  the  pattern  of 
correlations,  it  appeared  that  the  first  dimension  described 
sectors  with  many  climbing  and  descending  flights,  requ'ring  the 
application  of  many  different  ATC  functions  or  procedure  These 
sectors  also  typically  contained  varying  numbers  of  facilities 
requiring  frequent  interfacility  coordination.  The  second 
dimension  was  defined  primarily  by  the  mix  of  aircraft  types  (VFR, 
IFR,  etc.) .  A  third  vector  in  the  space  was  related  to  the  number 
of  departing  and  arriving  aircraft. 

Table  B2  shows  the  property  vectors  for  the  TMU  staff.  The  pattern 
of  correlations  for  TMU  staff  is  more  complex  because  the  MDS  space 
was  three  dimensional  (table  B2) .  For  these  data,  the  first 
dimension  combined  number  of  departure  and  arrival  transitions, 
radio  coverage,  and  intersecting  flight  paths,  while  the  second 
dimension  incorporated  number  of  departing  and  arriving  aircraft, 
mix  of  aircraft,  and  number  of  military  requirements.  There  were 
no  clear  correlations  with  the  third  MDS  dimension,  but  there  were 
several  other  property  vectors.  The  first  three  additional 
property  vectors  were  not  parallel  with  the  primary  MDS  dimensions. 
These  three  property  vectors  corresponded  with  weather,  airline 
hubbing,  and  multiple  functions.  For  the  TMU  data,  a  fourth  vector 
combined  number  of  facilities,  amount  of  climbing  and  descending 
traffic,  and  size  of  airspace. 


Table  BI.  Regression  Results  for  Multidinensional  Scaling 
Solutions  for  Airspace  Procedures  Specialists 


PROPSRTY  VICTOR 

r2 

8Z6M. 

DZMSHSZOV  BBTA 
WIZGHTS 

CLIMBING/DESCENDING  FLIGHTS 

.95 

.0005 

l(-.78) 

MULTIPLE  FUNCTIONS 

.94 

.0009 

1(“.96) 

AIRCRAFT  MIX 

.92 

.0019 

2(-.87) 

NO.  OF  FACILITIES 

.91 

.0027 

l(-.79) 

COORDINATION 

.90 

.0029 

l(-.83) 

NO.  DEPARTING/ARRIVING  AIRCRAFT 

.90 

.0035 

l(-.68)  t  2(-.59) 

Table  B2.  Regression  Results  for  Multidimensional  Scaling 
Solutions  for  Terminal  Management  Unit  Staff 


PROPERTY  VICTOR 

IBM 

8Z6M. 

OZMEVaZOV  BBTR 
WBZ6RT8 

MO.  DEPARTING/ARRIVING  AIRCRAFT 

.70 

■Lim 

2(-.71) 

AIRCRAFT  MIX 

.67 

.0007 

2(-.80) 

AIRLINE  HUBBING 

.64 

.0013 

3(-.61)  i  2(-.57) 

MEATHER 

.62 

.0018 

3(-.58)  £  2(.49) 

MO.  OF  FACILITIES 

.60 

.0026 

l(-.54)  £  2(-.58) 

CLIMBIMG/DESCENDING  FLIGHTS 

.59 

.0028 

1(-.51)  £  2(-.59) 

STARS  £  SIDS 

.58 

.0037 

l{-.63) 

FREQUENT  MILITARY  REQUIREMENTS 

.57 

.0045 

2(-.61) 

RADAR/RADIO  COVERAGE 

.54 

.0070 

1(.73) 

INTERSECTING  FLIGHT  PATHS 

.54 

.0071 

l(-.73) 

MULTIPLE  FUNCTIONS 

.54 

.0072 

l(-.50)  £  3{-.47) 

SIZE  OF  AIRSPACE 

.52 

.0094 

1(.53)  £  2(.53) 

APPENDIX  C 


WORD  SHAPE  SORTING  TEST  (WSST)  RESULTS 


HSST  scores  for  the  FPLs,  APSs,  and  TMU  personnel  ranged  from  6  to 
59  (shown  in  table  Cl)  .  Across  these  groups,  31  percent  of 
participants  scored  20  or  below  (indicating  a  verbal/analytic 
problem  solving  style) ;  47  percent  fell  between  20  and  40  (showing 
no  strong  preference);  and  22  percent  scored  above  40  (indicating 
a  spatial/global  style) . 

The  WSST  scores  for  different  specialization  areas  were  averaged  to 
obtain  a  mean  (M)  for  each  group.  A  test  of  the  group  means  using 
Analysis  of  Variance  (ANOVA)  did  not  indicate  significant 
differences  between  North  (M  =  33.4),  South  (M  »  31.2),  East  (M  » 
31.4),  West  (M  -  32.8),  Central  (M  «  27.0),  Developmentals  (M  - 

33.6) ,  and  Airspace  Procedures /TMU  staff  (M  *  29.33),  F(6,  29)  * 
0.96,  p  «  .996.  The  mean  times  required  to  complete  the  WSST  also 
did  not  vary  between  specialization  areas.  The  results  (in 
seconds)  were:  North  (M  =  236.2),  South  (M  *  436.4,  East  (M  * 
219.2),  West  (M  »  355.0),  Central  (M  ==  291.0),  Developmentals  (N  « 

187.6) ,  and  Airspace  Procedures /TMU  staff  (M  «=  279.2),  F(6,  29)  * 
1.07,  p  -  .402. 

Product ->moaent  correlations  were  calculated  between  WSST  scores, 
INDSCAL  weirdness  values',  and  collapsed  MDS  participant  weights 
for  the  FPLs  in  the  main  study.  The  purpose  of  this  analysis  was 
to  determine  whether  or  not  thinking  style  accounted  for 
differences  in  emphasis  on  the  common  stimulus  space  dimensions,  as 
reflected  by  weirdness  values  and  collapsed  INDSCAL  weights.  No 
significant  correlations  were  found  between  the  WSST  score  and 
either  weirdness  or  weight  space  values.  (The  correlation  of  the 
WSST  score  with  weirdness  was  -0.20.  The  WSST  score  and  collapsed 
weight  correlation  was  0.20.) 


1  A  Miasure  of  how  close  s  subject's  MDS  scores  came  to  the  others  in  the  group 
or  shared  stimulus  space. 


Table  Cl.  Word-Shape  Sorting  Test  Scores  for  PPL,  APS, 

and  TKU  Participants 


CENTRAI. 

FPL 

CENTRAL 

FPL 

EAST 

FPL 

EAST 

FPL 

EAST 

FPL 

EAST 

FPL 

365  _ 26 

172  _ 

272  39 


10  _  EAST _  FPL _ 231  _ IJ 

11  EAST  DEVEL.  34 


12 

EAST 

DEVEL. 

203 

31 

13 

EAST 

DEVEL. 

259 

42 

14 

EAST 

DEVEL. 

314 

25 

IS 

EAST 

DEVEL. 

162 

36 

16 

NORTH 

FPL 

177 

39 

17 

NORTH 

PPL 

256 

12 
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